Gitlab access tokens not working anymore after upgrade to 16.x

Written by Claudio Kuenzler - 2 comments

Published on July 28th 2023 - last updated on July 31st 2023 - Listed in Linux Git

---

After upgrading a local Gitlab CE installation from 14.x to 16.x I noticed that access tokens didn't work anymore for pulling a private repository:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible:my-super-secret-token@gitlab.example.com/infrastructure/ansible.git
fatal: unable to access 'https://gitlab.example.com/infrastructure/ansible.git/': The requested URL returned error: 500

During the Gitlab upgrade path, which involved multiple version upgrades, I tested git pull both with login credentials and access tokens:

• 14.6.1 (initial version): both working
• 14.9.5: both working
• 14.10.5: both working
• 15.0.5: both working
• 15.1.6: both working
• 15.4.6: both working
• 15.11.13: both working
• 16.2.1: login working, access tokens not working

The access token (ansible) is still showing up in the repository settings in the now upgraded Gitlab:

But "something" seems to have changed concerning the token handling and using this token now results in an internal server error (HTTP 500). 

Creating a new repository access token

I created a new access token (ansible2023) with the same permissions (role: developer, scope: read_repository) as the old token (ansible):

Both tokens now show up in the list:

Note: I noticed the "Expires" column has changed for the original ansible token from "never" to "in 11 months"... ?

By using the new token (ansible2023) and its secret, the repository pull was working again:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible2023:my-new-fancy-secret@gitlab.example.com/infrastructure/ansible.git
From https://gitlab.example.com/infrastructure/ansible
 * branch            HEAD       -> FETCH_HEAD
Already up to date.

By the way, if you assign the "guest" role (instead of "developer"), this won't be enough to pull the repository and you get the following error:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible2023:my-new-fancy-secret@gitlab.example.com/infrastructure/ansible.git
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab.example.com/infrastructure/ansible.git/': The requested URL returned error: 403

In the Gitlab access logs, the pull can be seen with a HTTP 200:

xxx.xxx.xxx.xxx - - [28/Jul/2023:11:15:01 +0200] "GET /infrastructure/ansible.git/info/refs?service=git-upload-pack HTTP/1.0" 401 279 "" "git/2.30.2" -
xxx.xxx.xxx.xxx - ansible2023 [28/Jul/2023:11:15:01 +0200] "GET /infrastructure/ansible.git/info/refs?service=git-upload-pack HTTP/1.0" 200 164 "" "git/2.30.2" -
xxx.xxx.xxx.xxx - ansible2023 [28/Jul/2023:11:15:01 +0200] "POST /infrastructure/ansible.git/git-upload-pack HTTP/1.0" 200 145 "" "git/2.30.2" -

Where did the breaking change come from?

Of course the big question is: Why did it work before and since the upgrade to 16.x not anymore? The cause seems to be a change mentioned in the deprecation and removals documentation:

Non-expiring access tokens

    Announced in GitLab 15.4 (Sep 2022)
    Removal in GitLab 16.0 (May 2023) (breaking change)

Access tokens that have no expiration date are valid indefinitely, which presents a security risk if the access token is divulged. Because access tokens that have an exipiration date are better, from GitLab 15.3 we populate a default expiration date.
In GitLab 16.0, any personal, project, or group access token that does not have an expiration date will automatically have an expiration date set at one year.

It doesn't say whether or not the previously active tokens will become immediately invalid, but this seems to be the case given what I just experienced.

Update: Old token working again

Now a few days after the Gitlab upgrade and a weekend with food poisoning (ugh!), I tested the old access token once more. And interestingly it worked:

ck@ansible:/pub/ansible$ cd /pub/ansible && /usr/bin/git pull https://ansible:my-super-secret-token@gitlab.example.com/infrastructure/ansible.git
From https://gitlab.example.com/infrastructure/ansible
 * branch            HEAD       -> FETCH_HEAD
Already up to date.

So to me it looks that the upgrade to Gitlab 16.x triggered the following internally once Gitlab was started:

• Detect old access tokens with a expiry date set to "never"
• Migrate/adjust these tokens to an expiry date of ~1 year (in 11 months, as seen in the screenshot)
• During that migration the token doesn't work and an internal server error (500) is shown to the git client
  

Add a comment

Show form to leave a comment

Comments (newest first)

ck from Switzerland wrote on Jul 31st, 2023:

Manuel, thanks for the comment. Yes, I saw that too. But as you can see from my upgrade experience, the previously used ansible token stopped working after the upgrade to Gitlab 16.x. I noticed that the expiry date was adjusted from never to in 11 months however this existing token did not work anymore after the upgrade. Only new tokens would work again. According to a Gitlab employee, I talked to in the meantime, this could be a bug.

---

Manuel Kraft from wrote on Jul 31st, 2023:

Hey Claudio,

Thank you for posting your upgrade documentation.

Just want to share with you and your readers, that the token DO NOT get enforced to stop working after upgrading to 16.0 / 16.x .

The personal access tokens will receive an expiration of 1 year in the future after you upgrade and batched background migrations are run.

As GitLab is open source, and if you need to see code to be confident, the value is set to 365.days.from_now here

Ref:
https://gitlab.com/gitlab-org/gitlab/-/blob/eb0f0465ff1f55f1347b981f2502a8e4d4fe6fe4/lib/gitlab/background_migration/cleanup_personal_access_tokens_with_nil_expires_at.rb#L10

Regarding project / group tokens: Under the hood Project and Group access tokens are also Personal Access tokens; the user they belong to is a "bot" user rather than a real human. So this new expiration applies to all 3 types of tokens.

Ref: https://docs.gitlab.com/ee/development/internal_users.html

---