Even if you can't persuade HMRC that it is your home , you will pay no CGT anyway.
You say you bought for £298,500 in 2005 .You can add purchase costs onto this eg the survey fee , stamp duty (3% ?), Solicitors costs etc so say another £10k making a base cost of £308,500.
If you sell for £320,000 then you can deduct selling fees eg Foxtons fee (2% + VAT ?) Solicitors fees etc so say £8.5k meaning a net sale price of £311,500.
Your gain is £311,500 - £308,500 = £3k.
Your annual CGT exemption is £10k so no tax to pay.
Simples