Fix Broken External Links

@sselhorn i've started putting this together... my thinking is contributors can "claim" a batch at a time?

We just need to provide a few pointers. I wonder if I maybe try and attack batch 1 if I will find enough variety to comment on the different scenarios?

Love to hear your thoughts- thanks a mill

Here's my attempt at thinking out loud while reviewing the 1st batch. Can I/we boil this down to some basic steps/suggestions/guidelines?

- public/charts/charts/globals.html:
  https://www.haproxy.com/blog/haproxy/proxy-protocol/: link has moved permanently to 'http://www.haproxy.com/blog/use-the-proxy-protocol-to-preserve-a-clients-ip-address/'

I've skipped this one for the moment as it's in a different repo. I think just this one slipped through?

- public/ee/administration/auth/oidc.html:
  https://casdoor.org/docs/integration/gitlab/: 404

This was a nice perplexing one... Hitting the URL did indeed return a 404, but still loaded the page AOK. It wasn't immediately apparent what was going on, so I found the gitlab link on the navigation menu and followed it. Very subtle... but the link was missing the trailing forward slash... e.g. https://casdoor.org/docs/integration/gitlab

- public/ee/administration/geo/disaster_recovery/index.html:
  https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal: link has moved permanently to 'https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal'

This one was reasonably straightforward, but the link was to an anchor, so I had to visit the page to ensure the anchor was still valid.

- public/ee/administration/geo/replication/object_storage.html:
  https://cloud.google.com/storage-transfer/docs/: link has moved permanently to 'https://cloud.google.com/storage-transfer/docs/overview'

This one was as simple as updating and testing.

- public/ee/administration/get_started.html:
  https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000803379: 403
  https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=334447: 403
  https://support.gitlab.com/hc/en-us/requests/new: 403

I'm not seeing the same response. I wonder if this is false positive?

- public/ee/administration/integration/mailgun.html:
  https://app.mailgun.com/app/account/security/api_keys: link has moved permanently to 'https://login.mailgun.com/login'
  https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks/: link has moved permanently to 'https://www.mailgun.com/blog/product/a-guide-to-using-mailguns-webhooks/'

I suspect https://app.mailgun.com/app/account/security/api_keys requires authentication before it can be viewed, so that is the reason for the redirection not because the page has moved.

https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks/ on the otherhand has simply moved and needed updating.

- public/ee/administration/monitoring/performance/grafana_configuration.html:
  https://grafana.com/docs/grafana/latest/reference/export_import/: link has moved permanently to 'http://grafana.com/docs/grafana/v7.5/dashboards/export-import/'
  https://grafana.com/docs/grafana/latest/installation/: link has moved permanently to 'http://grafana.com/docs/grafana/next/setup-grafana/installation/'

Here, following the links did indeed redirect to the addresses reported BUT the first was a specific v7.5 doc, which I didn't think we wanted, so followed another link through to the latest doc to get the correct URL.

Similarly, the latter link was to the "next" version docs, but we wanted the latest version, so next needed updating with latest.

- public/ee/administration/monitoring/performance/performance_bar.html:
  https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp: link has moved permanently to 'https://web.dev/critical-rendering-path-measure-crp/'

This was a straightforward update.

- public/ee/administration/packages/container_registry.html:
  https://www.docker.com/blog/registry-v1-api-deprecation/: 404

~~I had to google this one to find the new URL.~~ Google turned up something similar looking, but "we" don't think it is quite the same thing, so have removed the link.

- public/ee/administration/postgresql/external.html:
  https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users: link has moved permanently to 'https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-create-users'

The redirect was correct, and I just had to check the anchor was still valid.

- public/ee/administration/raketasks/github_import.html:
  https://docs.github.com/en/rest/reference/rate-limit: link has moved permanently to 'https://docs.github.com/en/rest/rate-limit'

This was a simple switch.

- public/ee/administration/raketasks/storage.html:
  https://support.gitlab.com: link has moved permanently to 'https://support.gitlab.com/hc'

I'm unsure whether we should update this one or now (we may even want to consider adding support.gitlab.com to our exclusion list? My concern is that the redirection is based on the client (e.g. I get taken to https://support.gitlab.com/hc/en-us

Here's the resulting MR: !91946 (merged)

changed the description

marked the checklist item Batch 1 as completed

mentioned in commit 8a66fd7a

mentioned in merge request !91946 (merged)

@leetickett Nice! Your comments are exactly what the tech writers go through. Which is why we have such a backlog, I think.

I did want to call out that we have four repos where we have to make fixes:

The main GitLab repo
Omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab
Runner: https://gitlab.com/gitlab-org/gitlab-runner
Charts: https://gitlab.com/gitlab-org/charts/gitlab

Also, if you go to this issue, search for Check for broken external links. and read some of the info there, to see if it helps.

I do like the idea of putting it in batches! One other idea might be to sort it by page where the broken link is, if it's not already. That would be going above and beyond though.

@marcel.amirault You know more about our broken links than I do. Would you be able to help Lee with some of his review questions?

Fortunately the output is already grouped by page (if/where multiple broken links exist on a single page)

Great point about the different repos, it makes no sense to have a batch which targets multiple repos- i'll adjust a few of them quickly now

I think something weird may have happened and the links are all duplicated too... under /public/blah and also /public/public/blah?

I need to double check that tomorrow, and if so I can immediately cut the list in half

Thanks for all the support @sselhorn

@leetickett Thanks for this. Not sure what's happening with the duplication there. It appears we're publishing the docs site twice!

See:

As for the batches, we could also split it up based on type of error, like: https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md#remote-tasks. The highest priority is 404s.

@leetickett the problem with duplication appears to be with: https://gitlab.com/gitlab-org/gitlab-docs/-/blob/d3ce55082b3df44d81eb577d89b5a66a9573d707/scripts/minify-assets.sh.

I can reproduce locally.

CC @axil @marcel.amirault @sarahgerman

I've raised: gitlab-docs!2900 (merged)

@leetickett I think we've fixed the duplication, so I've started another pipeline to update the list: https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/582428022

I think using batches is great, but the 403's and 404's can be a little challenging. As a first step, how about we create batches of just the links that are redirecting? These links are really easy to fix, and also really easy for the TW team to review. Taking care of those would also drastically shrink the output of the link test job, which will make it easier for us to address the harder links. WDYT?

Really appreciate it, thanks @marcel.amirault

That sounds like a great first step

@leetickett Just manually triggered the link test now: https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/2691142591

I GitPod @gtsiolis

I have literally no idea what i'm doing with linux/bash whatever it's called... but come up with this so far:

bundle exec nanoc check external_links > external_links
grep -v -E "403|404" external_links > filtered_external_links
awk '$1=="["{print filename; print $9 " -> " $15} $1 ~ /public/{filename = $1}' filtered_external_links

Which i'm sure can be improved greatly, but it's a good start... output looks like:

gitpod /workspace/gitlab-docs (main) $ awk '$1=="["{print filename; print $9 " -> " $15} $1 ~ /public/{filename = $1}' filtered_external_links
public/charts/charts/globals.html:
https://www.haproxy.com/blog/haproxy/proxy-protocol/: -> 'http://www.haproxy.com/blog/use-the-proxy-protocol-to-preserve-a-clients-ip-address/'
public/ee/administration/integration/mailgun.html:
https://app.mailgun.com/app/account/security/api_keys: -> 'https://login.mailgun.com/login'
public/ee/administration/raketasks/storage.html:
https://support.gitlab.com: -> 'https://support.gitlab.com/hc'
public/ee/administration/redis/replication_and_failover.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/redis/replication_and_failover.html:
https://redis.io/topics/security: -> 'https://redis.io/topics/security/'
public/ee/administration/redis/replication_and_failover_external.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/redis/troubleshooting.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/reference_architectures/10k_users.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/reference_architectures/10k_users.html:
https://redis.io/topics/security: -> 'https://redis.io/topics/security/'
public/ee/administration/reference_architectures/25k_users.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/reference_architectures/25k_users.html:
https://redis.io/topics/security: -> 'https://redis.io/topics/security/'
public/ee/administration/reference_architectures/3k_users.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/reference_architectures/3k_users.html:
https://redis.io/topics/security: -> 'https://redis.io/topics/security/'
public/ee/administration/reference_architectures/50k_users.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/reference_architectures/50k_users.html:
https://redis.io/topics/security: -> 'https://redis.io/topics/security/'
public/ee/administration/reference_architectures/5k_users.html:
https://redis.io/topics/sentinel: -> 'https://redis.io/topics/sentinel/'
public/ee/administration/reference_architectures/5k_users.html:
https://redis.io/topics/security: -> 'https://redis.io/topics/security/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/1adf30bef7e34ceba9efa97c4470417b: -> 'https://azure.com/e/1adf30bef7e34ceba9efa97c4470417b/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/8f618711ffec4b039f1581871ca6a7c9: -> 'https://azure.com/e/8f618711ffec4b039f1581871ca6a7c9/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/69724ebd82914a60857da6a3ace05a64: -> 'https://azure.com/e/69724ebd82914a60857da6a3ace05a64/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/de3da8286dda4d4db1362932bc75410b: -> 'https://azure.com/e/de3da8286dda4d4db1362932bc75410b/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/0dbfc575051943b9970e5d8ace03680d: -> 'https://azure.com/e/0dbfc575051943b9970e5d8ace03680d/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/3f973040ebc14023933d35f576c89846: -> 'https://azure.com/e/3f973040ebc14023933d35f576c89846/'
public/ee/administration/reference_architectures/index.html:
https://azure.com/e/72764902f3854f798407fb03c3de4b6f: -> 'https://azure.com/e/72764902f3854f798407fb03c3de4b6f/'
public/ee/administration/troubleshooting/linux_cheat_sheet.html:
https://about.gitlab.com/support/statement-of-support.html: -> 'https://about.gitlab.com/support/statement-of-support'
public/ee/api/openapi/openapi_interactive.html:
https://swagger.io/specification/: -> 
public/ee/architecture/blueprints/consolidating_groups_and_projects/index.html:
https://design.gitlab.com/objects/merge-request: -> 'https://design.gitlab.com/objects/merge-request/'
public/ee/architecture/blueprints/database_scaling/size-limits.html:
https://about.gitlab.com/handbook/engineering/development/enablement/database/: -> 'https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/'
public/ee/architecture/blueprints/database_testing/index.html:
https://ops.gitlab.net: -> 
public/ee/architecture/blueprints/database_testing/index.html:
https://about.gitlab.com/handbook/engineering/development/enablement/database/: -> 'https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/'
public/ee/ci/cloud_services/google_cloud/index.html:
https://cloud.google.com/iam/help/workload-identity/attribute-mapping: -> 'https://cloud.google.com/iam/docs/workload-identity-federation#mapping'

Discussing this while pairing with @zillemarco

We can see there is already some ruby in the project, so we could leverage ruby to make life easier santizing/preparing the output.

We could then create the issue directly, either using the gitlab ruby gem or as we probably only need to call one endpoint, a raw http request.

Do you think it's worth us moving forward with this, or is it just as easy for someone to run the CI job or nanoc task in GitPod every few weeks and spin up an issue by hand?

Thanks!

@sarahgerman Would it be a big deal to add this ruby gem to the gitlab-docs repo?

@leetickett

Do you think it's worth us moving forward with this, or is it just as easy for someone to run the CI job or nanoc task in GitPod every few weeks and spin up an issue by hand?

Yes please. Any effort to automate these monthly tasks, like the ones listed here, especially to create issues for the community, would be greatly appreciated.

@sselhorn In general we're moving towards using more JavaScript and less Ruby in this project. The proposed next version of the site will use a JavaScript site generator instead of a Ruby one, and so our supporting scripts will also be JavaScript/node.js.

The existing link checker comes from nanoc (the Ruby site generator). The next version of the site will not use nanoc, and so it will also not use this link checker.

I would advise against investing time in nanoc-specific features and instead recommend that we try a different tool for flagging broken links. We can make this change on the current site, with little risk (especially since our current link checker is kinda bad). I'd imagine a more modern tool will provide us with better output out-of-the-box. Just doing a quick search, this one looks nice: https://github.com/stevenvachon/broken-link-checker#readme

As for generating an issue automatically (very cool idea!), this looks like the most popular node.js GitLab API client: https://github.com/jdalrymple/gitbeaker#readme

Thanks so much @sarahgerman

@leetickett Here is your answer--

Hi @sarahgerman

Just for context, how far are we from not using nanoc for the site generation? I tried to have a quick look to see try and see that but I couldn't really tell

Anyway, even if we stick to nanoc, we should be able to replace it only for the test that checks for external links by using a JS tool like the one you suggested and running it against the generated site pages, right? Thanks basically what nanoc does now if I understood it right

Hey @zillemarco! I'd guess the move away from nanoc will take several months, but we haven't gotten into it enough to have a solid timeline yet. We're working on a few other things and also waiting for the release of nuxt.js version 3. If you're curious, there is a proof-of-concept of docs on nuxt over here: https://gitlab.com/oregand/gitlab-docs-v2

You're right about the nanoc link checker -- it's only used on that job that runs the test. There are (minimal) docs for it here: https://nanoc.app/doc/testing/

I'd be happy to help review MRs or test a JavaScript solution for this, feel free to ping me whenever. But if sticking with nanoc makes more sense for now, that's fine too, I'd just recommend keeping custom code minimal.

Thanks a lot for the explanation @sarahgerman

waiting for the release of nuxt.js version 3

I never heard of that and it sounds super interesting

If you're curious, there is a proof-of-concept of docs on nuxt over here: https://gitlab.com/oregand/gitlab-docs-v2

That also looks very interesting!

But if sticking with nanoc makes more sense for now, that's fine too, I'd just recommend keeping custom code minimal.

Let's see what @leetickett also thinks about this, but I think that if, in the end, we'll move towards removing the Ruby code from the docs, we might as well start from somewhere Replacing Ruby on one part of the pipeline, which seems to be "quite" unrelated from the actual site, might be worth taking a look I mean, if it doesn't complicate things too much

@leetickett @zillemarco @sarahgerman @sselhorn

I'd imagine a more modern tool will provide us with better output out-of-the-box. Just doing a quick search, this one looks nice: https://github.com/stevenvachon/broken-link-checker#readme

Just as an FYI, I added markdown-link-check (https://github.com/tcort/markdown-link-check#readme) to gitlab-development-kit (gitlab-development-kit!2382 (merged)) a while ago. There, we don't check external links (https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/markdown-link-check-config.json) but it supports it.

At the time, it didn't support anchors properly: gitlab-development-kit!2382 (comment 815969797). I'm not sure if nanoc external link checker also supports anchors on external links.

There has been some anchor-related work on the tool recently: https://github.com/tcort/markdown-link-check/releases, but I haven't circled back to check if anchors work properly now. Given I originally started off with 3.9.0, there's a chance it works now.

A benefit of this tool is also a downside. It's really great to be able to run a link checker on source Markdown, because you don't need any sort of HTML transformation first. Of course, links not contained in Markdown are missed. For example, links in: https://gitlab.com/gitlab-org/gitlab-docs/-/blob/e777f2d81f6b17110e1775bc53f6304d4a953fae/content/index.erb would be missed. gitlab-development-kit docs aren't transformed and so this wasn't a problem for that project.

changed the description

mentioned in commit 4c60027b

mentioned in merge request gitlab-docs!2900 (merged)

added [deprecated] Accepting merge requests [deprecated] good for new contributors documentation labels

@leetickett I moved this issue into the main GitLab repo so it shows up in the list for contributors. I think it makes sense to have it in that repo, since all the fixes will be in that repo (for the most part).

Thanks for working on this @leetickett! We've removed the Seeking community contributions label to avoid having multiple people working on the same issue.

removed [deprecated] Accepting merge requests label

@leetickett I've been poking at a few of these broken links as part of the monthly maintenance tasks. Some are an easy fix (like the missing trailing slash), and others need more digging since the target has been rewritten, moved, deleted, no longer valid/supported etc.

As of today I've got 2 MRs for this and probably will spin up a couple more soon:

Awesome thanks

My plan is to rerun very close to Hackathon and share for community contribution

@leetickett I'll xlink any new MRs as well just so the loop is happening

mentioned in merge request !93034 (merged)

mentioned in merge request !92949 (merged)

mentioned in issue technical-writing#639

Fix Broken External Links

Charts (WIP MR)

GitLab

Omnibus

Operator (WIP MR)

Runner (WIP MR)

Designs

Child items ...

Activity