Commit Graph

77 Commits

Author SHA1 Message Date
José Romildo
a15e083372 httpTwoLevelsUpdater: fix variable default value 2022-10-01 12:27:09 -03:00
José Romildo
0b9c39e819 gitUpdater: fix variable default value 2022-10-01 12:27:09 -03:00
José Romildo
19a4717846 directoryListingUpdater: init
Update a package looking for available versions in an html page
2022-10-01 12:27:09 -03:00
José Romildo Malaquias
e78c23cf5b
Merge pull request #193248 from romildo/upd.generic-updater
generic-updater: more flexible with name, pname, version and attr path, and cleanups
2022-09-30 09:19:21 -03:00
José Romildo
f15117a874 generic-updater: more flexible with name, pname, version and attr path
- This information is availabe from environment variables defined by
maintainers/scripts/update.nix

- Renamed the shell script to generic-update-script.sh

- Add a new optional argument (representing the package name) to the
shell script

- The version lister is called with a new optional
argument (representing the package attribute path)
2022-09-27 23:41:53 -03:00
Jan Tojnar
e628b43a9c common-updater-scripts: fix silent error on 404
When TOFU was unable to download the file, there would be no hash
in the fetch log, causing the grep to fail. Since the script
is set to errexit, it would terminate the processing without
any output. Let’s instead print the fetch log.
2022-09-27 02:19:25 +02:00
Jan Tojnar
31c2154465 common-updater-scripts: Use custom fake temporary hash
Some update scripts (e.g. sublime4) already use all-zeroes hashes
for resetting a version when updating multiple sources.
When `update-source-version "${attrPath}" 0 "${lib.fakeSha256}"`
was executed, u-s-v would not be able to detect that the hash
changed and would fail with the following message:

    Failed to replace temporary source hash of '…' to the final source hash!

The update script worked when we tested it because change detection itself
was broken until recently: https://github.com/NixOS/nixpkgs/pull/190554
2022-09-16 18:36:01 +02:00
Jan Tojnar
ee1795cc87 common-updater-scripts: Only look for URL when replacing it
This would do a pointless slow `nix-instantiate` call for a feature
that is rarely used – initially, it was added for Firefox updates
186de9ca9e
but now not even that uses it.

Additionally, this would break stuff like `cargoDeps`,
which lack the url attributes.
2022-09-09 22:31:28 +02:00
Jan Tojnar
17bc96df08 common-updater-scripts: Fix replacement failure detections
Previously, we only made `sed` back up the version replacement.
This meant that `cmp` would already recognize the files
as changed before replacing hash and the other values,
and the error would not be printed.

Let’s always make the `sed` create a backup so that we can
detect success of each situation. This will no longer
allow us to revert to the original version on failure
but the updated file should be tracked in git anyway.
2022-09-09 22:31:28 +02:00
Jan Tojnar
9137e671a2 update-script-combinators.copyAttrOutputToFile: init
Useful for storing generated files into Nixpkgs repo as a part of an update script to avoid a build time dependency.
2022-09-07 14:23:28 +02:00
Jan Tojnar
7d6b5d171a update-script-combinators: experimental init
The sequence combinator will be useful for updating packages consisting of multiple sources.
2022-09-07 14:23:28 +02:00
Alyssa Ross
88441f3e44 unstableGitUpdater: fix inaccurate comments
It used to use src.url, but when that was changed the comments weren't
updated.

Fixes: 7aae279ad9 ("unstableGitUpdater: fix updating fetchzip-based sources")
2022-09-04 08:59:51 +00:00
OPNA2608
0276d2fcbb common-updater-scripts/update-source-version: Add flag required for nix-command 2022-07-07 23:01:35 +02:00
José Romildo
cb9a33497f generic-updater: fix nix edit command line
Recently `nix edit` works only with `--extra-experimental-features nix-command`
2022-07-02 08:17:00 -03:00
Jan Tojnar
7aae279ad9 unstableGitUpdater: fix updating fetchzip-based sources
a67950f20b added `url` attribute
from `fetchurl` and therefore also from `fetchzip`.
We previously relied on `url` from fetchgit-based fetchers
to find the repo URL but now it will just return tarballs
in the case of `fetchFrom{GitHub,GitLab}`.

Let’s add an attribute to `fetch{git,FromGitHub,FromGitLab}`
to expose a repo URL consistently.
2022-05-24 19:12:12 +02:00
Kira Bruneau
f14a57c95b gitUpdater: update comment for url 2022-04-23 15:30:56 -04:00
Kira Bruneau
ef57e2fb8d common-updater-scripts/list-git-tags: prefer src.meta.homepage 2022-04-23 15:30:56 -04:00
Sergei Trofimovich
219ca58452 gitUpdater: add explicit url parameter to specify a git tree for tags
That way gitUpdater could be used to extract most recent published
releases for .nix files that don't use git directly to define builds.

An example is iproute2 package which does not have a single place
to lookup most recent release.
2022-04-01 18:10:45 +01:00
José Romildo
005ac63a19 httpTwoLevelsUpdater: init 2022-02-21 13:01:49 -03:00
José Romildo
ba59355e81 gitUpdater: init
- Add the 'gitUpdater' helper function to update git based packages, using the
'genericUpdater' function.
- Rework argument passing to the `list-git-tags' and 'list-archive-two-level-versions' scripts.
- Replace 'genericUpdater' plus 'list-git-tags' by 'gitUpdater'
2022-02-21 13:01:49 -03:00
Jan Tojnar
97276fe650 unstableGitUpdater: Allow using stable versions
It is important to keep the version as parsed by `builtins.parseDrvName`
monotonic for packages that often switch between stable and unstable versions,
for nix-env to be able to update them reliably.

Let’s optionally use the version format described by RFC 107:
https://github.com/NixOS/rfcs/pull/107
The downside is that it requires fetching
more commits to be able to determine the latest tag.
2022-02-18 17:26:46 +01:00
Jan Tojnar
3985dde04e unstableGitUpdater: Use single derivation for all branches
This will reduce the number of build when updating a large number of packages at the same time
(e.g. when updating all packages of a maintainer).
2022-02-18 17:26:45 +01:00
Jan Tojnar
30ec84fba3 unstableGitUpdater: Fix support for fetchFromGit{Hub,Lab}
When `fetchFromGitHub` uses `fetchzip` instead of `fetchgit` internally,
which is the most common use case, there is no `src.url` attribute
so the update will fail.

Let’s fix that by falling back to `src.meta.homepage`,
which these fetchers set to the repository URL.
2022-02-18 07:59:02 +01:00
Jan Tojnar
d9ff6d90aa common-updater-scripts: Silence outPath check
It is expected to fail in repos not using flakes-compat (like Nixpkgs).
2021-12-31 07:06:27 +01:00
Jan Tojnar
d9dea8d2a9 common-updater-scripts: allow using different src attribute
Useful for packages with multiple sources.
2021-12-31 06:54:42 +01:00
Samuel Gräfenstein
17d18904e2
common-updater-scripts/update-source-version: Fix on Nix 2.4 2021-10-03 20:24:12 +02:00
Samuel Gräfenstein
8ae7cd8a93
common-updater-scripts/*: /bin/sh -> /usr/bin/env bash 2021-10-01 18:42:18 +02:00
Samuel Gräfenstein
9a1bf524f5
common-updater-scripts/list-git-tags: don't print commands 2021-09-29 16:04:21 +02:00
Attila Lendvai
6c876bac61 unstable-updater.nix now understands a branch argument 2021-06-17 16:54:46 +02:00
Jan Tojnar
3771de8ae0
common-updater-scripts: Support attribute lookup in flake
In flakes, packages are not exposed directly but instead they are declared
inside “packages” or “legacyPackages” output under their host platform.
flake-compat reflects this.

Let’s look for an attribute also in these outputs if the direct lookup fails.
2021-04-24 04:52:48 +02:00
Jan Tojnar
38e20a3242
common-updater-scripts: Replace flake source by local path
When update-source-version is run in a repo with flake-compat,
it would find the files in Nix store.

Let’s replace them with the local path of the repository.
2021-04-24 03:42:01 +02:00
Jan Tojnar
3c781e6d19
common-updater-scripts: handle default.nix without arguments
So that the script can be used outside of nixpkgs.
For example, the default.nix recommended by flake-compat does not take arguments.
2021-04-24 03:41:33 +02:00
Jan Tojnar
d788c091b6
common-updater-scripts: remove lib dependency
So that the script can be used outside of nixpkgs.
2021-04-24 02:50:53 +02:00
Ben Siraphob
e03c068af5 treewide: makeWrapper buildInputs to nativeBuildInputs 2021-02-19 20:09:16 +07:00
Pavol Rusnak
a6ce00c50c
treewide: remove stdenv where not needed 2021-01-25 18:31:47 +01:00
Pavol Rusnak
90f7338112
treewide: stdenv.lib -> lib 2021-01-24 01:49:49 +01:00
Matej Urbas
ca9e2a3c3b pkgs.common-updater-scripts: update-source-version name singular 2021-01-09 10:08:52 +00:00
Sandro Jäckel
0b940b9977 scripts/mark-broken: fix path 2020-11-25 13:08:27 -08:00
Francesco Gazzetta
3f20319cd1 unstableUpdater: init 2020-11-21 11:28:45 +01:00
Francesco Gazzetta
4d8e0f9b77 common-updater-scripts: add --rev to update-source-version
Adds a --rev=<revision> parameter to the script that makes it possible
to explicitly specify a new revision.
Useful to update unstable packages, where the version and revision may
be independent.
2020-11-21 11:26:46 +01:00
Jonathan Ringer
0544a7f672 scripts/mark-broken: improve
This improves on the previous verison of this script.

Previously it only accepted one attr, and required
explicit passing of the "broken" value.

This script is meant to be used to mark failing hydra builds as broken in the meta attrs
To use the script, you should pass the list of failing attrs as arguments to the script.

Example: `cat failing-attrs | xargs ./pkgs/common-update/scripts/mark-broken`

Generating a list of failing attrs: (this should be improved at a later date)
  - Go to the most recent hydra evaluation with all builds completed
  - Select the "builds still failing" tab
  - Highlight and select all packages, should be prefixed with `nixpkgs.`
  - Use regex and editor foo to leave only the attr names
  - Use the above example command to then execute the script

OTHER NOTES:
  - The `denyFileList` and `denyAttrList` will likely need to be updated slightly
    to align with the conventions used in nixpkgs at execution time
  - Any attrs which failed for any reason will be written to `failed-marks.txt`.
    Those attrs will likely need manual attention as disablement will likely be conditional.
2020-10-30 23:11:36 -07:00
Jan Tojnar
d351cea9f3
common-updater-scripts: add --print-changes flag
Printing the changed file and new version can be used to commit the changes to git.
2020-09-19 17:02:54 +02:00
worldofpeace
d12c11e30c treewide: use nix-update updateScript for packages I maintain 2020-07-29 12:33:39 -04:00
José Romildo Malaquias
8bceb7c876 genericUpdater: add attribute argument to ignore some versions 2020-05-04 14:33:03 -03:00
worldofpeace
05d1546060
Merge pull request #85337 from petabyteboy/feature/mark-broken
common-updater-scripts: add mark-broken script
2020-04-24 19:16:41 -04:00
José Romildo Malaquias
7d84c7cd36 common-updater: change default development version check to false 2020-04-16 08:06:56 -03:00
José Romildo Malaquias
5aad6a361d common-updater: add update script to all-packages 2020-04-15 23:02:33 -03:00
Milan Pässler
b74be134db common-updater-scripts: add mark-broken script 2020-04-16 03:01:26 +02:00
José Romildo Malaquias
827a6619eb common-updater-scripts: add scripts to help update packages
- updateScript:
  A nix expression that can be used in passThrough to update a package

- list-git-tags:
  A shell script to list available tags in a git repository

- list-archive-two-level-versions:

  A shell script to list available versions in a web site in two
  levels: there is a page listing the available major.minor versions,
  and for each of them there is another page listings the patch level
  versions major.minor.patch.

  It is suitable for Xfce packages for instance.

How the updater works:

1. collect the available versions from the source repository (using a
script given as argument)

2. print the collected versions (for debugging)

3. (optionally) apply some transformation to the collected versions to
make them compatible with the versions used in nixpkgs (for instance,
tags in the Xfce git repository may be prefixed with the package name,
and the prefix need to be removed)

4. sort the available versions in decreasing order

5. choose the first version that pass validation:
   - check if the version may be a development version

   - if the version IS unstable, skip it and give a warning about
   skipping a development version (for debugging)

   - if the version COULD BE unstable, take it and give a warning
   about taking a potential development version (for debugging)

   - if the version IS stable, take it

6. update the package version and checksum in its nix expression

7. print the git commands for adding the modified files and for
committing the changes
2020-04-15 09:45:25 -03:00
Jan Tojnar
09a4a051e8
common-updater-scripts: Fix replacing SRI hashes
SRI hashes (base64 encoded) can contain + sign which is a special character
in extended regular expressions so it needs to be escaped.
2020-02-20 07:18:36 +01:00