There are more than 100,000 secondary residences in Paris for a population of 2 million So 22,000 for London is nothing! Sorry!!!
You don't need to speak French; look at the numbers in central districts: 8e (Champs-Elysees), 4e (Marais)...
Well it is very funny you talk about Ile St-Louis. My studio flat (which I expect to sell to buy a flamboyant flat in SE London ) is actually on that plan (but in the "mainland").
That part of Paris (the "Iles" - the other is where Notre-Dame is -, Marais - North - and Latin Quarter - South -)... I would compare it to Leicester Square or Soho. So they don't really attract the Arab/Chinese/Russian which will head to the West (around Champs-Elysees, which would be the local "Mayfair") but maybe wealthy artists and so on..
Prices are not that high. My flat is about 10-12,000 EUR/sqm now. Yeah some top locations can command higher prices. For instance at Place des Vosges, Place Dauphine... if you have good views it can go up to 20-25,000 EUR/sq (very rare). Also new construction which is EXTREMELY rare there (just some old buildings completely rebuilt).
There are less and less residents in those areas indeed but the secondary residences is just one of the reasons.
To me, the main reason is tourism. Tourism in Paris (like London) seems to never stop growing (now you have all those Chinese, Russian, Arabs... that you didn't have 15 years ago). In those thin streets in those areas (like the Marais, Quartier Latin)... there is literally no space. So every weekend (no matter the season) they are PACKED. Well that is very annoying for a resident. You literally step out of your house to be in a flow and crowd of tourists. You have to queue in the metro station to go through the gates (no joke). My building has an historical feature (so it is featured in guides) and I see continuously tourists taking pictures of it (I can see it from my kitchen) since it seems to be in some itinerary with local guides. Sometimes it is funny, sometimes it is annoying.
Second thing... because of tourists... retail becomes totally oriented for them (vicious circle). So it is not very cool if they close a supermarket to open who knows what (an art gallery, some flagship store, some antiquarian...). And even local shops are more expensive. My local PAUL (the bread chain that you have in London) is about 10-20% more expensive than others in areas just about 2 or 3 metro stations. The reason? (I asked a manager once). Tourists (which count for most of the customers there) are willing to pay more and rentals are more expensive. I am lucky enough to have a large supermarket next door (probably the largest in that whole area) but even there is full of tourists which means that they gradually orient it more to "take away" products rather than a real supermarket (and of course it is ridiculously expensive... it is like living in Piccadilly and having Whole Foods as your supermarket)
And then finally tourism means AirBNB. Those small apartments in central Paris (mine is 300 sqft) are just PERFECT for AirBNB (really nobody cares about size; it is all about location when renting in AirBNB, and the location is top). Why rent full-time or live there when you can rent a 300sqft studio for 70 EUR a night? Just make your numbers! You cannot rent that apartment for much more than 1000 EUR. But even if you rent it for 15 days... you already break even compared to a full-time rental (and they don't pay income tax usually on AirBNB ).
Of course in Ile St-Louis that is even crazier because you are in an island where (I think) there is not even a proper supermarket. It is cool because it is very quiet at night (and you are in the middle of the city), but it is definitely not handy (not to mention if you have kids, a car...it just doesn't work).
So yeah it is a vicious circle. Many people "give up" and move to areas just about 500 meters or 1 km. that are more livable and prices are lower (not only property, but shops). Because otherwise you end up living in a tourist trap with all the inconveniences: you street flocked with tourists; crossing Chinese tourists in your elevator going to their AirBNB flat (really nothing against them, but it doesn't make a lot of "community" if you see the point) and having more retail choices to buy €1,000 shoes than a bottle of milk or some meat in a supermarket.