March Site Update Progress

It has been about a month now since I last posted a site update, and there has been a whole lot of stuff happening behind the scenes during that time.

Here’s the current Trello status:

The ‘done’ column actually goes down quite a long way.

A lot of the work has been on the Climb migration which I briefly talked about here.

The gist of this is that the old way that the Climbs pages worked involved a whole bunch of steps to display the data. All of the climb info comes from Strava. And as the Climbs page quite proudly states:

I was really proud of this number, when I first published it.

But over time I’ve realised that I’d rather have gone for Quality over Quantity.

Here’s such an example:

For whatever reason, this segment has been removed from Strava. Well, removed, or made private.

When I look at climbs like this, I tend to see fairly low numbers – less than 500 people have ridden the climb.

Ultimately I wanted to drop these ‘junk’ climbs, and that solved two problems:

1. Gets rid of old / deleted climbs that are cluttering up the site
2. Cuts down on the total list I need to re-fetch the address for

Here’s the query I use:

select count(strava_id) from segments
where enabled = true
          and activity_type = 'Ride'
          and athlete_count >= 1000
          and average_grade > 0
          and effort_count >= 1000
          and total_elevation_gain >= 5

Previously I think I was doing more than 30 riders, and 50 efforts.

That cuts down the total number of climbs from 2.8 million to 779,470. Still a big number.

Now, double that number because I need to fetch the address for each one of those climbs, but I need to do it twice – once for the English / Westernised address, and once again for the address nationalised to that country.

If you look at the main Climb page right now, you will see what I mean:

This is one of my biggest face palms I ever made on this site.

I didn’t actually realise what I’d done there until far too late.

What I’m going to have to do now – and am in the process of doing – is getting each climb again with the two different addresses. I can only get 1 per second, and I did the maths on Saturday and it’s 17 days if running non-stop to fetch the whole 1.45 million.

Between now and then, the site is in a bit of a pickle.

The online database is slightly out of sync with the database I use when working on the site on my computer. This is a problem as if I then do what’s called a ‘static build’ of the site, which makes it nice and fast – but also, as the name implies, quite rigid – it breaks a ton of pages because some of my local routes are now pointing at the Westernised versions, not the old / nationalised versions.

Basically you might see a bunch of these errors right now when using the climbs.

More face palms… but to be honest, it’s been in bad shape for so long now, I think it’s short term pain for longer term gain. Like an FTP test.

What’s Changed?#

Aside from me banging on about all this crap I’m sure you don’t much care about (but it is cathartic for me to write), it’s probably more interesting to see what’s changed (and what is changing).

Right now things look fairly similar.

That is a little deceptive, however, as behind the scenes a heck of a lot has changed.

Each climb page now has the latest posts in the footer:

That’s not a massive change, but I would like people to read the stuff I write, so I made it a bit more prominent.

The Country landing pages are currently very slow.

This is … obviously not an improvement.

However, as above, they have had to be made ‘dynamic’ for the moment because of the issues above about the addresses.

That means on each page load, the entire page contents have to be retrieved from the database. This is very slow for a bunch of reasons. Normally I pre-render all these pages ahead of time, so they load almost instantly.

Again, this will resolve in 17 days … sigh.

This should also get rid of that horrible problem above where Scotland and Wales show twice.

Under each of these Country pages is what is called a State page. So in the example above, England would be a State.

That’s because that’s the way addresses come out of the address lookup service:

{
    place_id: 310523245,
    licence: 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',
    osm_type: 'way',
    osm_id: 410024287,
    lat: '37.593326180499346',
    lon: '-122.38454849230534',
    category: 'highway',
    type: 'secondary',
    place_rank: 26,
    importance: 0.10000999999999993,
    addresstype: 'road',
    name: 'Trousdale Drive',
    display_name: 'Trousdale Drive, Burlingame Village, Burlingame, San Mateo County, California, 94030, United States',
    address: {
      road: 'Trousdale Drive',
      neighbourhood: 'Burlingame Village',
      town: 'Burlingame',
      county: 'San Mateo County',
      state: 'California',
      'ISO3166-2-lvl4': 'US-CA',
      postcode: '94030',
      country: 'United States',
      country_code: 'us'
    },
    boundingbox: [ '37.5930395', '37.5934859', '-122.3848612', '-122.3843711' ]
  }

It doesn’t matter where you look in the world, and obviously using the USA was a terrible example here – but was the one happening as I looked at that very second – and the address always has a state key.

Anyway, it doesn’t really matter on what everyone sees – there is no mention of ‘State’ to site users. Except here. Now you know.

Those pages, again, are currently very slow. Not quite as slow as Country, but they, too, are usually pre-rendered / static, and right now have to be dynamic. I’m probably absolutely slaughtering my Google rankings.

Next we have the County pages.

Eagle eyed viewers will have noticed the regional maps have disappeared:

If you don’t recall, or never saw it, here’s how it should have looked:

And you’re probably thinking it was better with the map! Why remove it?

It turns out it had broken in some other recent update … somehow:

No matter how long you waited, it never loaded.

Anyway, it has only gone temporarily. Until I can fix it.

Here’s how it currently looks, if enabled:

Which, I think you will agree, isn’t quite as attractive as when it used to work.

To be fair, it took me ages to implement that first map. And in order to use it in the new site I have to re-do it all, pretty much from scratch. Given how much other stuff I had / have to do, it got pushed down the todo list. I decided I would disable it and get something out to show some progress – but that was before I realised that broke a bunch of other stuff, as above.

Can’t win.

This map was also to be found in the Suburb page – so from the list above, if you clicked in to “Cleen” for example, you would see a more zoomed in map, on wherever in the world Cleen lives. Somewhere in County Tyrone, Northern Ireland it would appear.

And then the revamped climb view:

The big change is that veloviewer is now optional.

I reckon most people will want veloviewer to always load – so just hit the toggle and it always will.

So why do it?

Because Google hated it.

When the veloviewer loads, it makes the page contents jump about. That’s called a Cumulative Layout Shift and having that really tanks your Google rankings.

Combine that with all my other ham fisted slapdasherry and it’s amazing Google wanted anything to do with me.

So now it’s ‘off’ by default (aka, for Google), but for the average human who wants it, it can be enabled easily.

I want to do more with the climb pages, but for the moment I have enough on my plate.

Big Search Update#

Perhaps the biggest reason to make all these changes was to do with search.

This innocent looking little box… it contained a lot of problems behind the scenes.

The big one was that in order for search to work, I had to expose the entire database in some way. I did this using a thing called GraphQL, which worked really well, but was a big overhead both in terms of me knowing how to use it optimally, and perhaps far more importantly, securely.

I was never 100% convinced I had secured everything down, and always expected to wake up one day and found some devious bugger had changed a ton of stuff without me every realising. It shouldn’t be possible, but as I didn’t write the server – it was a plugin type thing – I could never be absolutely sure.

Getting rid of that was a big weight off my mind.

And that was all made possible because I switched from NextJS 13 to NextJS 14 behind the scenes. Again, technically mumbo jumbo, but it solved a big problem I had.

It created a few, too. Nothing’s perfect.

But here’s the new search page:

If you’re on slow internet, the Countries, States, and Counties sections will all show little ‘skeleton’ loaders whilst matching search results are found in the database and then displayed on the page.

It’s far more obvious with the individual climbs lookup, which takes ages anyway.

When the climbs are finally found, it replaces the skeleton blocks with the results:

All of that happens without any page reloads.

And you can now link to results easily because the URL looks like this:

https://bikeclimbs.com/climb/search?q=beacon

I’d love to make the climb lookup faster – and actually having ‘only’ 700,000 or so to look through will be far faster than the 2.8m it currently has to crawl.

Even so, 700,000 or 2.8m are very small numbers to computers so the fact it takes ages is not great. I need to look into it further. There will be ways to speed it up, no doubt.

But now it is far less likely to throw 500 errors:

That was another big win.

Further improvement will come in time.

Still To Come#

The big ticket items I’m working on right now (or when I get the chance) are:

1. Fixing up the regional maps
2. Find my location
3. MyWhoosh workout search

Of all of those, #1 seems to be the hardest.

Workout Search should hopefully be fairly easy to do.

I have some ideas around ‘find my location’ – it’s the feature I’ve wanted on here for the longest time. And a few days back I had a smart idea about how to solve it. Now I just need to find a few hours to have a bash at implementing it.

Anyway, those are my burdens to bear. Hopefully you’ve enjoyed this peak behind the curtain.