Jump to content
House Price Crash Forum
Dr. Statham

Improving Cacheing To Speed Up Hpc, Especially Forums.

Recommended Posts

Hi,

I've been looking into why accessing the forums has slowed down recently. It's been blamed on the `news ticker' and I think that was part of the problem, but it's not all of it and probably got the sole blame because it was a visible change.

I've made alterations to the machine here to workaround what I think are problems with the HPC site and find the forum pages load much faster. Hopefully, if the HPC admins agree, they can change things at the HPC end resulting in faster forums for everyone, and less bandwidth and server load for HPC.

Many HPC web pages pull in a bunch of Javascript programs by specifying their URL, e.g. http://www.housepricecrash.co.uk/forum/ loads

from HPC and a couple of others from outside.

I'm finding that my browser, Firefox, is loading the HPC Javascript programs on every page access, and in turn Squid, which is the HTTP proxy that Firefox is accessing the Internet through, is also loading them rather than returning them from its cache of previously accessed URLs.

This results in the forum page starting to load, followed by a slow down as the loading of the Javascript programs kicks in, which appears as two delays after the page begins to be rendered, both near the top of the page.

Squid isn't cacheing the Javascript because the HTTP headers returned by HPC's Apache web server don't give sufficient information to determine if the URL is out of date; if Squid has no means of finding if it's out of date then it must assume it is and fetch it every time.

Here's the HTTP headers returned for ips_menu.js above.

  • HTTP/1.1 200 OK
  • Date: Sun, 21 May 2006 12:06:07 GMT
  • Server: Apache/2.0.53 (Fedora)
  • X-Powered-By: PHP/4.3.11
  • Transfer-Encoding: chunked
  • Content-Type: text/html; charset=ISO-8859-1

It ships me the Javascript program, but the headers don't include any of Last-Modified, Expires, Cache-Control, or Etag so there's nothing to identify this particular version of the program, e.g. that it was last modified at noon yesterday, or tell me how long this copy is good for, e.g. it expires at noon tomorrow. Consequently, the next time I load a web page referring to it I can't think `no need to fetch it because it doesn't expire until noon tomorrow', or say to HPC `only ship it to me if it's changed since noon yesterday since that's the last version I got'.

Another problem with the headers is Content-Type says the Javascript program has a MIME-type of text/html. This is wrong since it isn't HTML. It happens to work anyway because the browser treats it as Javascript regardless but the correct MIME-type is application/x-javascript.

Now it may be that these headers are deliberately missing in order that changes to the Javascript programs have an immediate impact on users but I doubt it. And that would just explain why Last-Modified and Etag are missing; Expires could still be present set to `noon tomorrow' so all users fetch fresh versions then when the admins are around to field any problems.

I've told Squid to pretend that lots of the HPC Javascript is cacheable for an hour or so and find that after the first couple of forum web pages the whole thing gets much nippier and Squid reports that it's returning cached versions to Firefox rather than fetching them each time. Even users that currently don't have any problems with forum speed will possibly notice an improvement and it will cut the amount of data HPC has to ship, and the load on the server.

If anyone else is running Squid locally they may be interested in the refresh_pattern configuration options I ended up with.

  • refresh_pattern ^ftp: 1440 20% 10080
  • refresh_pattern ^gopher: 1440 0% 1440
  • refresh_pattern ^http://www\.housepricecrash\.co\.uk/scripts/javascript/..*\.js$ 60 100% 120
  • refresh_pattern ^http://www\.housepricecrash\.co\.uk/forum/jscripts/..*\.js$ 60 100% 120
  • refresh_pattern ^http://www\.housepricecrash\.co\.uk/forum/style_images/.*/folder_js_skin/..*\.js$ 60 100% 120
  • refresh_pattern . 0 20% 4320

Cheers,

Dr. Statham,

Consultant Radiologist.

Share this post


Link to post
Share on other sites

Hi,

...technostuff...

Cheers,

Dr. Statham,

Consultant Radiologist.

er - thanks for that, doc.

i think.

anyhoo - more interestingly is there going to be another series of "green wing"?

Share this post


Link to post
Share on other sites

Above my head, but sounds like it's worth looking at. Any advice on how we can actually implement this speed up locally or do we need to mess around with the Java SDK?

Oh, and watch out for green dwarves, DrS....

TLM

Share this post


Link to post
Share on other sites

Hello Monsieur le Monde,

Any advice on how we can actually implement this speed up locally or do we need to mess around with the Java SDK?

By locally do you mean locally at home or locally at the HPC server? As far as I'm aware there is no Java involved. Perhaps you're getting confused between the two due to their both using the name of the most populous island in the world?

Oh, and watch out for green dwarves, DrS....

Well, thank you for your concern, Monsieur le Monde, but I think you'll find I'm perfectly prepared for them with my trusty Ardea melanocephala.

Regards,

Dr. Statham,

Consultant Radiologist.

Edited by Dr. Statham

Share this post


Link to post
Share on other sites
Guest wrongmove

Some good points there Dr. Statham. We've made a couple of changes based on this. Do you think it's faster now even with the ticker?

Not for me :(

Share this post


Link to post
Share on other sites

Hello Webmaster,

Some good points there Dr. Statham. We've made a couple of changes based on this. Do you think it's faster now even with the ticker?

Do you mean is it faster now you've re-enabled the `ticker' with or without my local modifications to decline to load afresh the Javascript on each access? Without my Squid's refresh patterns it's a very slow thing. With them, it's nippy but then scroller.js and scrollernews.js are both matched by the patterns so are only downloaded once, id est the ticker is just another case of a stale, unvalidateable, HTTP reply.

What changes have been done? I still see similar results to before, viz

$ unset http_proxy$ wget -q -s -O - \>	 http://www.housepricecrash.co.uk/forum/jscripts/ipb_global.js |> sed '/^$/q'HTTP/1.1 200 OKDate: Mon, 22 May 2006 22:37:31 GMTServer: Apache/2.0.53 (Fedora)X-Powered-By: PHP/4.3.11Connection: closeContent-Type: text/html; charset=ISO-8859-1$

To quell the natives it may be circumspect to confiscato secundo the ticker whilst examination continues.

Regards,

Dr. Statham,

Consultant Radiologist.

Share this post


Link to post
Share on other sites

Hello Monsieur le Monde,

By locally do you mean locally at home or locally at the HPC server? As far as I'm aware there is no Java involved. Perhaps you're getting confused between the two due to their both using the name of the most populous island in the world?

Well, thank you for your concern, Monsieur le Monde, but I think you'll find I'm perfectly prepared for them with my trusty Ardea melanocephala.

Regards,

Dr. Statham,

Consultant Radiologist.

yup, I'm easily confused - tweaks to my local PC was what I meant.

Ta,

TLM

Share this post


Link to post
Share on other sites
Hello Webmaster,

Do you mean is it faster now you've re-enabled the `ticker' with or without my local modifications to decline to load afresh the Javascript on each access? Without my Squid's refresh patterns it's a very slow thing. With them, it's nippy but then scroller.js and scrollernews.js are both matched by the patterns so are only downloaded once, id est the ticker is just another case of a stale, unvalidateable, HTTP reply.

What changes have been done? I still see similar results to before, viz

$ unset http_proxy$ wget -q -s -O - \>	 http://www.housepricecrash.co.uk/forum/jscripts/ipb_global.js |> sed '/^$/q'HTTP/1.1 200 OKDate: Mon, 22 May 2006 22:37:31 GMTServer: Apache/2.0.53 (Fedora)X-Powered-By: PHP/4.3.11Connection: closeContent-Type: text/html; charset=ISO-8859-1$

To quell the natives it may be circumspect to confiscato secundo the ticker whilst examination continues.

Regards,

Dr. Statham,

Consultant Radiologist.

The ticker js now loads within the head tag which probably means that it's not being pulled everytime the page is loaded.

Share this post


Link to post
Share on other sites

Hello Webmaster,

The ticker js now loads within the head tag which probably means that

it's not being pulled everytime the page is loaded.

I'm no expert in HTML and Javascript, unlike radiology in which I'm a

consultant, but I'd have thought that placing the load of

scrollernews.js inside the HTML header just results in it executing

sooner, and doesn't imply anything about the caching of it by browser or

proxies.

It's all very well to form a hypothesis but risky to the patients to put it

into practice without testing. And the patients are revolting. It's quite

easy to examine the browser's requests. Here's Firefox accessing first

/forum/index.php?showforum=35 followed by /forum/index.php?showtopic=30514.

I've recorded some network packets with tcpdump(8). Both GET many Javascript

programs including scrollernews.js.

$ request()

> {

> perl -ln0777e 'print /GET [^\x0d]+?\bHTTP.+?\x0d\x0a\x0d\x0a/sg' "$@"

> }

$ listget() { cat -A "$@" | grep '^GET'; }

$ request pack0 | listget | cut -c -72

GET /forum/index.php?showforum=35 HTTP/1.0^M$

GET /scripts/css/forum_header.css HTTP/1.0^M$

GET /scripts/javascript/minmax.js HTTP/1.0^M$

GET /scripts/javascript/scroller.js HTTP/1.0^M$

GET /scripts/javascript/scrollernews.js HTTP/1.0^M$

GET /forum/jscripts/ipb_global.js HTTP/1.0^M$

GET /forum/jscripts/ips_menu.js HTTP/1.0^M$

GET /forum/style_images/1/folder_js_skin/ips_menu_html.js HTTP/1.0^M$

GET /forum/jscripts/ips_xmlhttprequest.js HTTP/1.0^M$

GET /pagead/ads?client=ca-pub-6655171571856784&dt=1148379004329&lmt=1148

GET /pagead/ads?client=ca-pub-6655171571856784&dt=1148379004329&lmt=1148

GET /forum/jscripts/ipb_global_xmlenhanced.js HTTP/1.0^M$

GET /forum/jscripts/dom-drag.js HTTP/1.0^M$

GET /forum/jscripts/ipb_forum.js HTTP/1.0^M$

GET /forum/style_images/1/f_norm_no_dot.gif HTTP/1.0^M$

GET /pagead/ads?client=ca-pub-6655171571856784&dt=1148379013673&lmt=1148

GET /adverts/opadvert.html HTTP/1.0^M$

$ request pack1 | listget | cut -c -72

GET /forum/index.php?showtopic=30514 HTTP/1.0^M$

GET /scripts/javascript/minmax.js HTTP/1.0^M$

GET /scripts/javascript/scroller.js HTTP/1.0^M$

GET /scripts/javascript/scrollernews.js HTTP/1.0^M$

GET /forum/jscripts/ipb_global.js HTTP/1.0^M$

GET /forum/jscripts/ips_menu.js HTTP/1.0^M$

GET /forum/style_images/1/folder_js_skin/ips_menu_html.js HTTP/1.0^M$

GET /forum/jscripts/ips_xmlhttprequest.js HTTP/1.0^M$

GET /pagead/ads?client=ca-pub-6655171571856784&dt=1148379051563&lmt=1148

GET /pagead/ads?client=ca-pub-6655171571856784&dt=1148379051563&lmt=1148

GET /forum/jscripts/ipb_global_xmlenhanced.js HTTP/1.0^M$

GET /forum/jscripts/dom-drag.js HTTP/1.0^M$

GET /forum/jscripts/ipb_topic.js HTTP/1.0^M$

GET /forum/uploads/av-2284.jpg HTTP/1.0^M$

GET /forum/uploads/av-1801.jpg HTTP/1.0^M$

GET /forum/uploads/av-3174.gif HTTP/1.0^M$

GET /forum/uploads/av-1880.jpg HTTP/1.0^M$

GET /pagead/ads?client=ca-pub-6655171571856784&dt=1148379063546&lmt=1148

$

And examination of pack1's GET of scrollernews.js shows that it was

being requested in entirety because pack0's reply contained nothing to

help cache it, id est nothing has improved since my first post

and things have worsened by re-instating extra un-cachable URLs to

download!

Regards,

Dr. Statham,

Consultant Radiologist.

Share this post


Link to post
Share on other sites
Guest wrongmove

What connection speed are you on?

Only 1 meg at home, with IE6 (hangs head in shame :P ). Works fine at work (10M with Firefox).

At home (where I use HPC most) it can take 10 seconds to load a long page of posts. Never used to be a problem.

Cheers, wrongmove.

Share this post


Link to post
Share on other sites
Guest wrongmove

Have you tried Firefox at home? Also what about the Lo-Fi version?

Tried the low-fi version :o

Running Firefox at home now - much better. Thanks for your help. :D

Share this post


Link to post
Share on other sites

Hello again Webmaster,

And examination of pack1's GET of scrollernews.js shows that it was

being requested in entirety because pack0's reply contained nothing to

help cache it, id est nothing has improved since my first post

and things have worsened by re-instating extra un-cachable URLs to

download!

A little onomatopoeic chiffchaff told me about *********************.com

and that it used similar forum software, so I put it to the test.

$ while read f; do

> HEAD "$f" |

> gi '^(content-type|last-modified|etag|expires|cache-control):'

> echo

> done

http://forum.*********************.com/jsc...s/ipb_global.js

ETag: "a60062-4a51-44108081"

Content-Type: application/x-javascript

Last-Modified: Thu, 09 Mar 2006 19:22:41 GMT

http://forum.*********************.com/menuscript.js

ETag: "33480dc-429-4412d327"

Content-Type: application/x-javascript

Last-Modified: Sat, 11 Mar 2006 13:39:51 GMT

http://forum.*********************.com/favicon.ico

ETag: "334811e-57e-441083b3"

Content-Type: image/x-icon

Last-Modified: Thu, 09 Mar 2006 19:36:19 GMT

http://forum.*********************.com/sty...ps_menu_html.js

ETag: "2bcc133-b2a-4346ab2d"

Content-Type: application/x-javascript

Last-Modified: Fri, 07 Oct 2005 17:06:53 GMT

$

It passed, ergo its HTTP replies are cachable.

Regards,

Dr. Statham,

Consultant Radiologist.

Share this post


Link to post
Share on other sites
Guest wrongmove

A little onomatopoeic chiffchaff told me about *********************.com

and that it used similar forum software, so I put it to the test.

Now you are sounding like tit !

:lol::lol::lol:

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.

  • 302 Brexit, House prices and Summer 2020

    1. 1. Including the effects Brexit, where do you think average UK house prices will be relative to now in June 2020?


      • down 5% +
      • down 2.5%
      • Even
      • up 2.5%
      • up 5%



×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.