Commit Graph

100 Commits

Author SHA1 Message Date
adisbladis
4a5f2bd6e8 stdenv/check-meta: Use bespoke type checking
Aka `checkMeta` goes brrr.

Using the module system type checking works OK & generates good error messages.
The performance of using it however is terrible because of the value merging it does being very allocation heavy.

By implementing a very minimal type checker we can drastically improve the performance when nixpkgs is evaluated with `checkMeta = true`.
2023-12-26 15:14:42 +13:00
adisbladis
1e66093cca
Merge pull request #269546 from adisbladis/stdenv-meta-no-intermediate-alloc
stdenv: Avoid allocating intermediate attrset when checking meta validity
2023-12-12 10:57:44 +13:00
adisbladis
9a0a097a94 stdenv: Avoid allocating intermediate attrset when checking meta validity
This is a small performance optimization. It should be impercetible to most.

Benchmarks:

- Before
``` json
{
  "cpuTime": 0.2777960002422333,
  "envs": {
    "bytes": 3832648,
    "elements": 189513,
    "number": 144784
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 50229344
  },
  "list": {
    "bytes": 655304,
    "concats": 3249,
    "elements": 81913
  },
  "nrAvoided": 218962,
  "nrFunctionCalls": 127718,
  "nrLookups": 40946,
  "nrOpUpdateValuesCopied": 1563978,
  "nrOpUpdates": 8542,
  "nrPrimOpCalls": 113032,
  "nrThunks": 329605,
  "sets": {
    "bytes": 29774864,
    "elements": 1824537,
    "number": 36392
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 235909,
    "number": 24432
  },
  "values": {
    "bytes": 9691392,
    "number": 403808
  }
}
```

- After
```
{
  "cpuTime": 0.2615779936313629,
  "envs": {
    "bytes": 3833832,
    "elements": 189661,
    "number": 144784
  },
  "gc": {
    "heapSize": 402915328,
    "totalBytes": 50212960
  },
  "list": {
    "bytes": 655304,
    "concats": 3249,
    "elements": 81913
  },
  "nrAvoided": 218814,
  "nrFunctionCalls": 127718,
  "nrLookups": 40798,
  "nrOpUpdateValuesCopied": 1563978,
  "nrOpUpdates": 8542,
  "nrPrimOpCalls": 113032,
  "nrThunks": 329457,
  "sets": {
    "bytes": 29765392,
    "elements": 1824093,
    "number": 36244
  },
  "sizes": {
    "Attr": 16,
    "Bindings": 16,
    "Env": 16,
    "Value": 24
  },
  "symbols": {
    "bytes": 235909,
    "number": 24432
  },
  "values": {
    "bytes": 9687840,
    "number": 403660
  }
}
```
2023-12-12 00:08:34 +13:00
adisbladis
3b13bd5c84 stdenv: Avoid some list allocations in check-meta when checking licenses 2023-12-12 00:03:20 +13:00
Artturin
249c6d04b1 check-meta: don't use rec
possible performance improvement
2023-11-14 16:00:42 +02:00
Artturin
0527676bd9 check-meta: don't use with 2023-11-13 23:47:25 +02:00
Artturin
215951e4c8 check-meta: Improve performance
See c3c31aa798
2023-11-13 23:43:03 +02:00
Robert Hensing
e143a933f6 check-meta.nix: Fix flake note
- These new-cli commands can be used with `-f`, in which case they're
  evaluated with pure evaluation disabled.
- Nix 2.4+ is not part of the condition; "flakes" is fully descriptive
  and more relatable.
- Don't suggest that it only enables this variable.
- Just don't say too much.
2023-10-31 12:44:36 +01:00
Emily Trau
2ad68a7d90 stdenv: factor out meta attr augmentation for reusability
stdenv: check if separating commonMeta from assertValidity fixes laziness issue

```
error: cannot coerce null to a string

at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/ofborg-evaluator-3/pkgs/development/coq-modules/coq-lsp/default.nix:34:60:

    33|     homepage = "https://github.com/ejgallego/coq-lsp";
    34|     changelog = "https://github.com/ejgallego/coq-lsp/blob/${defaultVersion}/CHANGES.md";
```
2023-05-01 20:18:11 +03:00
Adam Joseph
13507da345 check-meta.nix: fix self-contradictory error messages
See https://github.com/NixOS/nixpkgs/pull/222792#pullrequestreview-1356114111

You can't just `lib.filter _ lib.systems.all` -- that throws away
important information, leading to nixpkgs disagreeing with itself
like this:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is only supported on ... x86_64-linux but not on requested x86_64-linux, refusing to evaluate.
```

After:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is not available on the requested hostPlatform:
         hostPlatform.config = "x86_64-unknown-linux-musl"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mipsel-linux"
           "mips64el-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [
           {
             isStatic = true;
             parsed = { };
           }
         ]
       , refusing to evaluate.
```
2023-03-24 00:15:57 -07:00
Weijia Wang
b46c97e1dc stdenv: generalise showPlatforms 2023-03-23 23:47:11 +02:00
Weijia Wang
2a7e1e0228 stdenv: fix error with patterned platforms 2023-03-23 17:27:49 +02:00
Sandro
979c6c1fe4
Merge pull request #213780 from SuperSandro2000/check-meta-platform
stdenv: show supported and requested platforms when check meta fails
2023-03-23 15:01:34 +01:00
Felix Buehler
bc3d5934d7 treewide: use lib.optionals 2023-02-14 19:11:59 +01:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
John Ericson
6e4a1b18d9 meta.pkgConfigModules: Init convention
See docs.

Follow-up work:

- Existing packages should be converted

- `defaultPkgConfigPackages` should assert on `meta.pkgConfigModules`
  and let `tests.pkg-config` alone test the build results.

CC @sternenseemann

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-02-03 09:37:31 -05:00
Sandro Jäckel
0a8e692d29
stdenv: show supported and requested platforms when check meta fails 2023-01-31 14:43:47 +01:00
Adam Joseph
778419b9e6 Revert "lib/meta.nix: platformMatch: allow predicate functions"
This reverts commit b7d097438b.
2023-01-27 01:49:39 -08:00
Adam Joseph
b7d097438b lib/meta.nix: platformMatch: allow predicate functions 2023-01-22 00:27:19 -08:00
Adam Joseph
098c6b0bec check-meta(hasUnsupportedPlatform): use lib.meta.availableOn
`hasUnsupportedPlatform` was not updated with #37395, so it does not
understand attrsets in `meta.[bad]platforms`.  In particular,
attrsets in `meta.badPlatforms` will "fail open" and be ignored.

Let's use `lib.meta.availableOn` instead of duplicating its logic.

Thanks to @alyssais for [noticing][1].

[1][https://github.com/NixOS/nixpkgs/pull/194148#discussion_r990817610]

Co-authored-by: sternenseemann <sternenseemann@systemli.org>
2023-01-11 19:31:52 +00:00
Adam Joseph
607d59fa9e check-meta.nix: make non-source consistent with documentation
The documentation for `meta.sourceProvenance` in
`doc/stdenv/meta.chapter.md` says: "the `meta.sourceProvenance`
attribute should be a list containing one or more value..."

Let's update check-meta.nix to require that `meta.sourceProvenance` is
a list, as the documentation says, rather than a single element.

Adding two extra keystrokes `[` and `]` when filling out this field is
an insignificant burden for package authors, and being able to assume
that the `meta.sourceProvenance` field is always a list greatly
simplifies any code that acts on the value of this field.

Since `meta.sourceProvenance` was just merged a few hours ago now is
the easiest time to fix this: nobody is using the feature yet.
2023-01-01 18:21:11 -08:00
Naïm Favier
4af22aab8e
stdenv/check-meta: do deep type checks
Use a wrapper around `mergeDefinitions` to type-check values deeply, so
that e.g. `maintainers = [ 42 ];` is an error.
2023-01-01 14:10:42 +01:00
figsoda
ec8cb34358 treewide: fix typos 2022-12-17 19:39:44 -05:00
piegames
87d738e864
Merge #195120: check-meta.nix: fix checkMetaRecursively option 2022-10-16 15:33:16 +02:00
arcnmx
466fd1439f check-meta.nix: fix checkMetaRecursively option
In specific cases, combining the `checkMeta` and `checkMetaRecursively`
config options would result in `error: infinite recursion encountered`

fixes #193296
2022-10-13 14:25:57 -07:00
Robert Scott
d02ac63f4f stdenv/check-meta: fix support for NIXPKGS_ALLOW_NONSOURCE=1 2022-09-28 23:18:33 +01:00
aszlig
8f98a6d39b check-meta: Add isHydraChannel
This is needed in order to mark a certain derivation containing a Nix
expression tarball to Hydra so that it is recognised as a channel.

When I first got an evaluation error due to using this meta attribute, I
was under the impression that nobody outside of Vuizvui[1] is using this
feature and that we don't have any occurrence of isHydraChannel in
Nixpkgs.

However, when working around[2] the issue I assumed that it's not
something that should be included in Nixpkgs because we're not using it
there.

It turned out that my assumption was wrong and we *do* use the attribute
in Nixpkgs, namely via releaseTools.channel, which is similar to what
we're doing in Vuizvui.

Since we already include a bunch of undocumented attributes in
metaTypes, it only makes sense to add isHydraChannel as well since it's
actually documented in the Hydra documentation[3].

[1]: https://github.com/openlab-aux/vuizvui
[2]: https://github.com/openlab-aux/vuizvui/commit/e0685e81b3fdc43a272f0
[3]: 53335323ae/doc/manual/src/jobs.md (meta-fields)

Signed-off-by: aszlig <aszlig@nix.build>
2022-09-28 14:12:45 +02:00
piegames
6762de9a28 check-meta.nix: type checking changes
- Enable metadata checking by default, see https://github.com/NixOS/nixpkgs/pull/25304#issuecomment-298385426
- Check metadata before any other package issues, see https://github.com/NixOS/nixpkgs/issues/191124#issuecomment-1246523976
- Document that type checks only apply to the top level of nested values.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
2022-09-25 16:37:15 +02:00
Robert Schütz
38c776b679 stdenv/check-meta: support NIXPKGS_ALLOW_NONSOURCE=0 2022-06-13 19:08:17 +00:00
Sandro
4c3e9f091a
check-meta: fix comment (#175517)
* check-meta: fix comment

* Apply suggestions from code review
2022-06-02 01:38:57 +02:00
Robert Scott
5bb9bf4774 meta.sourceProvenance: inline hasSourceProvenance
it may be what the license handling code does, but it's confusing and not very useful

Co-authored-by: Adam Joseph <54836058+a-m-joseph@users.noreply.github.com>
2022-05-30 16:27:34 +08:00
Robert Scott
7906ea6d9d allowNonSourcePredicate: use example of categorical permissivity
Co-authored-by: Adam Joseph <54836058+a-m-joseph@users.noreply.github.com>
2022-05-30 16:27:34 +08:00
Robert Scott
095eb91533 meta.sourceProvenance: disallow string values
strings complicate reasoning about values and may not be needed with `sourceProvenance`

Co-authored-by: Alexander Foremny <aforemny@posteo.de>
2022-05-30 16:27:34 +08:00
Robert Scott
da9162f667 add mechanism for handling meta.sourceProvenance attributes
heavily based on patterns used by licenses infrastructure, so may
appear overengineered for its initial level of use
2022-05-30 16:27:34 +08:00
ckie
4def222ea4
stdenv/check-meta: add a "maintainerless" warning
This warning logs when a package has no maintainers. It will stay silent
if `meta.maintainers` is not set at all, only complaining when it is an
empty list. In the future a separate warning could be added to allow for
that stricter behavior. Or this warning could be changed.
2022-05-03 22:29:14 +03:00
ckie
3a34b6c820
stdenv/check-meta: add an eval warning option
This will be used in the next commit in this patch series.
2022-05-03 22:29:12 +03:00
ckie
5e420c2455
stdenv/check-meta: turn validity.valid into a str
This will allow for adding more validity types in the future, such as a
warning type. (which is in the next commit in this series)

This is NOT a breaking change because validity.valid is never exposed
outside of `stdenv.mkDerivation`.
2022-05-03 22:28:25 +03:00
Artturin
9f05fc6661 config.allowUnsupportedSystem: define as option 2022-05-02 20:39:43 +03:00
Artturin
9f473092f8 config.allowBroken: define as option 2022-05-02 17:20:44 +03:00
Artturin
1c49b81263 config.allowUnfree: define as option 2022-05-02 17:20:25 +03:00
Artturin
33cce15e42 treewide: remove meta.repositories
there's no documentation for meta.repositories and its not widely used
2022-03-24 23:56:14 +02:00
Naïm Favier
1ffdf02435
stdenv/check-meta: remove onlyLicenses check 2022-02-17 13:09:24 +01:00
matthewcroughan
7bea56b425 stdenv/check-meta: add note for Flake usage
Flake users that use a command like `nix build nixpkgs#hello` on a
broken/insecure package will not be able to use an environment variable
to override that behavior, unless they pass `--impure` to the command.

Co-authored-by: pkharvey <kayharvey@protonmail.com>
2022-02-01 13:23:46 -05:00
Felix Buehler
59c55f4558 update-walker: remove because unused 2022-01-26 21:46:59 +01:00
Artturin
40944bbab7 stdenv/check-meta: add maxSilent
Hydra supports it
https://github.com/NixOS/hydra/blob/master/src/hydra-eval-jobs/hydra-eval-jobs.cc#L172
2021-08-07 02:18:59 +03:00
Ana Hobden
acfddd576e stdenv: support mainProgram in meta
Support `mainProgram` as an attribute of `meta` for packages.

This is an attribute used by [`nix
run`](https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-run.html#description)
to customize the main program of a package.

For example, `pkgs.neovim` provides a `/bin/nvim` executable which users
would (almost certainly) prefer `nix run` to execute instead of a
non-existing `/bin/neovim`.

Signed-off-by: Ana Hobden <operator@hoverbear.org>
2021-04-25 21:19:49 -07:00
WORLDofPEACE
4b10920ed1
stdenv/check-meta: change to allowlist and blocklist (#114127)
* stdenv/check-meta: change to allowlist and blocklist

* Update pkgs/stdenv/generic/check-meta.nix

Co-authored-by: Graham Christensen <graham@grahamc.com>
2021-02-23 10:25:18 -05:00
Bernardo Meurer
129ec8a4a5
stdenv: remove mention of flashplayer (in comments) 2021-02-08 09:38:43 -08:00
Daiderd Jordan
788f61cf3e
Merge pull request #85545 from LnL7/meta-available-flags
meta: expose availability flags in derivation metadata
2020-11-28 18:57:47 +01:00
Arnout Engelen
f6650152bd
Promote allowUnfreePredicate in remediation message 2020-11-09 17:26:02 +01:00