Gitlab Release job using link from artifacts

Question

I am working on creating automated pipeline for GitLab. My goal is to build a release version of application (which works perfectly fine) and pack it into msi package (which works fine).

In the following code:

release-job:  # Valid release section within the job
stage: release
image: registry.gitlab.com/gitlab-org/release-cli:latest              
script:
  - echo "Running the release job."
dependencies:
  - packaging
needs:
  - packaging 
release:
  tag_name: TestRelease
  name: 'Release TestRelease'
  description: 'Release created using the release-cli.'
  assets:
    links:
      - name: Setup.msi
        url: $(< artifact_url.txt | sed 's/^\s*|\s*$//g')

I am trying to get the link from

artifact_url.txt

This is causing the error:

error="failed to create release: API Error Response status_code: 400 message: Validation failed: Links url can't be blank, Links url must be a valid URL" version=0.16.0

I already tried this version of the same script:

release-job:  # Valid release section within the job
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest                 
  script:
    -
     |
     echo "Running the release job."
     ARTIFACT_URL=$(sed 's/^\s*|\s*$//g; s/ //g' artifact_url.txt)
     ARTIFACT_URL=${ARTIFACT_URL:2}
     echo "Setup.msi download URL: $ARTIFACT_URL"
  dependencies:
    - packaging
  needs:
    - packaging 
  release:
    tag_name: TestRelease
    name: 'Release TestRelease'
    description: 'Release created using the release-cli.'
    assets:
      links:
        - name: Setup.msi
          url: $ARTIFACT_URL

I tried it even with quotes in "url" section and even without "$" sign. But always with the same error as I described earlier.

PS: I know that link in txt file is OK since I saw it and it works properly.

Can I get some help here please?

Thank you

EDIT

So after several days I have tried this:

release-job:  # Valid release section within the job
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest              
  before_script:
    - apk add jq
    - jq --version
    - > 
      ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")
      echo "Setup.msi download URL: $ARTIFACT_URL"
      ASSETS_LINK=$(echo "{}" | jq --arg name "Setup.msi" --arg url "$ARTIFACT_URL" '. + {name: $name, url: $url}')
      echo "Assets link JSON: $ASSETS_LINK"
  script:
    - >
      release-cli create --name "Release TestRelease" 
      --tag-name "TestRelease" 
      --description "Release created using the release-cli." 
      --assets-link "$ASSETS_LINK"
  dependencies:
    - packaging
  needs:
    - packaging 

I had to use apk add but for some reason I have restrictions and can download new packages only using nuget.

This is the result I get:

$ apk add jq fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/main: network error (check Internet connection and firewall) WARNING: updating and opening https://dl-cdn.alpinelinux.org/alpine/v3.18/community: network error (check Internet connection and firewall) ERROR: unable to select packages: jq (no such package): required by: world[jq]

EDIT2

Now I tried this:

release-job:  # Valid release section within the job
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest              
  script:
    - | 
      echo "Getting Artifact Link"
      ARTIFACT_URL=$(sed 's/^\s*|\s*$//g; s/ //g' artifact_url.txt)
      ARTIFACT_URL=${ARTIFACT_URL:2}
      echo "Setup.msi download URL - $ARTIFACT_URL"
      ASSETS_LINK="{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
      echo "Assets link JSON - $ASSETS_LINK"
      echo "Creating Release"
      release-cli create --name "Release TestRelease" --tag-name "TestRelease" --description "Release created using the release-cli." --assets-link "$ASSETS_LINK"
  dependencies:
    - packaging
  needs:
    - packaging 

This time it showed me correct url in console, but I am still getting this error:

And got the error saying this:

error="new CreateReleaseRequest: failed to parse assets: 1 error occurred:\n\t* invalid asset: "{\"name\":\"Setup.msi\",\"url\":\"https://code.com/bigproject/big-project/-/jobs/17979/artifacts/download?file=build_artifacts/Setup.msi\r\"}" invalid character '\r' in string literal\n\n" version=0.16.0

Answer 1

The GitLab release CLI expects a valid URL for the assets' links, but the variable substitution is not happening as expected in the YAML file.

Build/Package ┌──────────┐  Release Job  ┌────────────────────────────────┐
Stage         │          │──────────────►│                                │
              │ packaging│               │ 1. Fetch artifact_url.txt      │
              │          │◄──────────────│ 2. Extract URL and assign      │
              └──────────┘               │    to $ARTIFACT_URL            │
                                         │ 3. Create Release with         │
                                         │    Setup.msi link from         │
                                         │    $ARTIFACT_URL               │
                                         └────────────────────────────────┘

You need to make sure artifact_url.txt has been saved in the previous stage as a Job artifact. Then, in the next stage, you can use a script to read this URL and use the GitLab Release CLI to create the release with the asset link.

Example:

stages:
  - build
  - package
  - release

# This is an example job that might generate and save the artifact URL.
# It demonstrates the job artifact step
# Adapt it to your actual build/packaging process.
generate-artifact-url:
  stage: package
  script:
    - echo "https://example.com/path/to/setup.msi" > artifact_url.txt
  artifacts:
    paths:
      - artifact_url.txt
    expire_in: 1 hour

release-job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  before_script:
    - 'ARTIFACT_URL=$(cat artifact_url.txt)'
    - 'echo "Setup.msi download URL: $ARTIFACT_URL"'
  script:
    - >
      release-cli create --name "Release TestRelease" 
      --tag-name "TestRelease" 
      --description "Release created using the release-cli." 
      --assets-link "{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
  dependencies:
    - generate-artifact-url

In release-job, the before_script section is used to read the URL from artifact_url.txt into an ARTIFACT_URL variable. That variable is then used in the script section.

The dependencies keyword makes sure release-job waits for generate-artifact-url to complete and can access its artifacts.

---

Using your script, I got a really weird error:

failed to parse assets: 1 error occurred:\n\t* invalid asset: \"{\\\"name\\\":\\\"Setup.msi\\\",\\\"url\\\":\\\"https://code.com/bigproject/big-project/-/jobs/171/artifacts/download?file=build_artifacts/Setup.msi\\r\\\"}\" invalid character '\\r' in string literal\n\n" version=0.16.0

That means a problem with the JSON string passed to the --assets-link option of release-cli create: the error message mentions an "invalid character '\r' in string literal" suggests that the URL string you are constructing contains a carriage return character (\r).

That happens if artifact_url.txt has Windows-style line endings (CRLF, \r\n) instead of Unix-style line endings (LF, \n). When you read the URL from artifact_url.txt into the ARTIFACT_URL variable, if the file has Windows-style line endings, the \r character becomes part of the URL string, which is not valid in a JSON string literal.

You can modify the command that reads the URL from artifact_url.txt to explicitly remove any carriage return characters. Use tr -d '\r' to delete \r characters:

before_script:
  - 'ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")'
  - 'echo "Setup.msi download URL: $ARTIFACT_URL"'

Make sure your JSON string is correctly formatted, and all special characters are properly escaped. Since you are constructing a JSON string in the shell command, double-check that quotes and backslashes are correctly handled. If available in your environment, jq is a powerful tool for safely constructing JSON strings. It automatically handles special character escaping:

before_script:
  - 'ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")'
  - 'ASSETS_LINK=$(jq -n --arg name "Setup.msi" --arg url "$ARTIFACT_URL" \'{"name":$name,"url":$url}\')'
  - 'echo "Assets link JSON: $ASSETS_LINK"'

The all script becomes:

stages:
  - build
  - package
  - release

# Assume 'generate-artifact-url' job is defined elsewhere, which outputs 'artifact_url.txt'

release-job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  before_script:
    # Strip any carriage return characters from the URL
    - 'ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")'
    # Debug: Print the cleaned URL
    - 'echo "Setup.msi download URL: $ARTIFACT_URL"'
    # Use jq to construct the JSON string for assets-link safely
    # That requires jq to be available in the image; adjust your image if necessary
    - 'ASSETS_LINK=$(echo "{}" | jq --arg name "Setup.msi" --arg url "$ARTIFACT_URL" \'. + {name: $name, url: $url}\')'
    # Debug: Print the JSON string to be used in assets-link
    - 'echo "Assets link JSON: $ASSETS_LINK"'
  script:
    # Use the constructed JSON string in the release command
    - >
      release-cli create --name "Release TestRelease" 
      --tag-name "TestRelease" 
      --description "Release created using the release-cli." 
      --assets-link "$ASSETS_LINK"
  dependencies:
    - generate-artifact-url

---

ERROR: unable to select packages: jq (no such package)

NuGet is primarily a package manager for the Microsoft development platform, including .NET. It is not designed to install system packages like jq on Linux-based containers or environments directly.

The registry.gitlab.com/gitlab-org/release-cli:latest image, which is based on Alpine Linux (as indicated by your attempt to use apk), will not be able to utilize NuGet in the same way it uses apk to manage system packages. That limitation is because NuGet does not manage Linux system packages or software not specifically targeting the .NET platform.

Since installing jq via apk directly in your CI job is not feasible due to your restrictions, and using NuGet for this purpose is not applicable, you could try and create a custom Docker image that includes both the GitLab Release CLI and jq. You can build this image from a base image that is similar to registry.gitlab.com/gitlab-org/release-cli:latest, adding jq during the image build process. That way, your CI job uses this pre-prepared image and does not need to install anything on the fly.

FROM registry.gitlab.com/gitlab-org/release-cli:latest

# Install jq
RUN apk add --no-cache jq

# Verify jq installation
RUN jq --version

You would build this image and push it to a Docker registry accessible by your GitLab CI runners. Then, update your .gitlab-ci.yml to use your custom image:

image: your-custom-image:latest

If customizing the Docker image is not an option, another approach is to manually construct the JSON string needed for the --assets-link without relying on jq. For instance, you could replace the jq-based JSON construction with a straightforward shell command:

before_script:
  - > 
    ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")
    echo "Setup.msi download URL: $ARTIFACT_URL"
    ASSETS_LINK="{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
    echo "Assets link JSON: $ASSETS_LINK"

That method bypasses the need for jq by manually formatting the JSON string, which is a viable workaround for simple cases like this one.

---

With this approach without using jq, I think something does not fit in since echo shows this:

"ASSETS_LINK={"name":"Setup.msi","url":""} echo Assets link JSON: "

And there is this error:

failed to parse assets: 1 error occurred:\n\t* invalid delimiter for asset: \"\"\n\n" version=0.16.0

The main issue is that the ARTIFACT_URL variable is not being correctly populated or passed through to the ASSETS_LINK JSON string.

First, make sure the artifact_url.txt file is indeed being created in a previous job and properly passed as an artifact. Confirm that the artifacts and dependencies sections are correctly configured to pass artifact_url.txt to the release-job.

Then modify the command structure to make sure variable persistence and correct execution context, which can be especially important in CI environments where commands are executed in separate subshells:

before_script:
  - |
    ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")
    echo "Setup.msi download URL: $ARTIFACT_URL"
    ASSETS_LINK="{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
    echo "Assets link JSON: $ASSETS_LINK"
  script:
    - |
      echo "Creating release with asset link..."
      release-cli create --name "Release TestRelease" 
      --tag-name "TestRelease" 
      --description "Release created using the release-cli." 
      --assets-link "$ASSETS_LINK"

That would make sure the ARTIFACT_URL and ASSETS_LINK variables are defined and used within the same execution context, avoiding the issue where the ARTIFACT_URL value does not persist into the ASSETS_LINK variable.

If the issue persists, consider directly incorporating the asset link construction and the release command into a single command sequence to avoid potential issues with variable scope:

script:
  - |
    ARTIFACT_URL=$(cat artifact_url.txt | tr -d "\r")
    echo "Setup.msi download URL: $ARTIFACT_URL"
    ASSETS_LINK="{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
    echo "Assets link JSON: $ASSETS_LINK"
    release-cli create --name "Release TestRelease" 
    --tag-name "TestRelease" 
    --description "Release created using the release-cli." 
    --assets-link "$ASSETS_LINK"

Combining these steps into a single script section makes sure the variables are defined and used within the same shell instance, mitigating scope issues and making sure the URL is correctly incorporated into the ASSETS_LINK JSON string.

---

invalid character '\\r' in string literal\n\n" version=0.16.0

The error message indicates that the ARTIFACT_URL still contains a carriage return character (\r) at the end, which is causing the JSON parsing to fail. That can happen if the file artifact_url.txt was generated on or passed through a Windows environment, leading to Windows-style line endings (CRLF, \r\n).

To make sure all carriage return characters are removed, you can directly include a tr -d '\r' command in your pipeline to remove any \r characters from the ARTIFACT_URL. Here is how you could adjust your script:

release-job:  # Valid release section within the job
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest              
  script:
    - | 
      echo "Getting Artifact Link"
      # Use tr to delete carriage return characters
      ARTIFACT_URL=$(cat artifact_url.txt | tr -d '\r')
      echo "Setup.msi download URL - $ARTIFACT_URL"
      ASSETS_LINK="{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
      echo "Assets link JSON - $ASSETS_LINK"
      echo "Creating Release"
      release-cli create --name "Release TestRelease" --tag-name "TestRelease" --description "Release created using the release-cli." --assets-link "$ASSETS_LINK"
  dependencies:
    - packaging
  needs:
    - packaging 

So instead of using sed for various substitutions, and then attempting to slice the string (which seemed to be your intent with ARTIFACT_URL=${ARTIFACT_URL:2}), this version directly pipes the contents of artifact_url.txt through tr -d '\r'. That makes sure any carriage return characters are removed, which should address the problem indicated by the error message.

By focusing on the removal of carriage return characters, this adjustment simplifies how you are cleaning the URL. That approach ensures the URL is correctly formatted for JSON without introducing syntax issues caused by leftover carriage return characters.

Answer 2

the release section doesn't evaluate variables in the same way that the script section does. This limitation means that dynamic variables, such as those you're trying to use for the URL in the assets subsection, won't be processed as you expect.

There are 2 steps to workaround the issue you are facing

1. Use the script section to prepare your artifact_url.txt and to dynamically generate a configuration or script that includes the required URL.
2. Use the GitLab Release CLI in a command line manner within the script section, rather than relying on the release keyword in the .gitlab-ci.yml file.

sample code that you can use as a template to dynamically set URL variable:

release-job:  # Valid release section within the job
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  script:
    - echo "Running the release job."
    - ARTIFACT_URL=$(sed 's/^\s*|\s*$//g; s/ //g' artifact_url.txt)
    - echo "Setup.msi download URL: $ARTIFACT_URL"
    # Now use the release-cli directly with the prepared URL
    - >
      gitlab-release create --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG
      --description "Release created using the release-cli."
      --assets-link "{\"name\":\"Setup.msi\",\"url\":\"$ARTIFACT_URL\"}"
  dependencies:
    - packaging
  needs:
    - packaging