Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

[RRFC] Platform-Specific Dependencies #120

Unanswered
natewatson999 asked this question in General
Mar 28, 2020 · 4 comments · 4 replies
Discussion options

Motivation ("The Why")

Everybody has seen those warnings from an npm install command where it didn't install an optional dependency, probably because the dependency in question was OS-specific to an OS the installer wasn't running on. A common issue is a warning that fsevents wasn't installed on an OS that wasn't OSX. Furthermore, there are hundreds of packages that are Windows-specific.

Right now, these print as console warnings during the installation process. This shouldn't be a warning, because npm is behaving the way it's supposed to and the correct packages are being installed.

Packages should be allowed to specify that a particular dependency is only used on certain platforms and not others.

Furthermore, the current system can obscure problems. if fsevents is required on OSX in a package and not elsewhere, and it fails to install due to something going wrong on an OSX client, the OSX user and logs will still see that it was just an optional dependency failing to install, even though it was really a mandatory dependency on that platform.

Right now, package.json has devDependencies, dependencies, peerDependencies, bundledDependencies, and optionalDependencies as dependency types; and packages can specify the intended architectures and operating systems of the packages themselves. To fix this problem of warnings that aren't warning about something wrong, the dependency set should be extended to include "platformDependencies".

Example

Right now, chokidar specifies fsevents as an optional depdendency. This means that whenever chokidar is installed on something other than OSX, there is a warning printed. Allowing platform-specific dependencies would solve this issue.

As god-king-emperor StackOverflow has shown, this is a problem in practice: https://stackoverflow.com/questions/38021422/nodejs-npm-install-platform-specific-packages

How

Current Behaviour

Currently, platform-specific dependencies are just treated as optionalDependencies, as shown here: https://github.com/paulmillr/chokidar/blob/e1c37c6207b0b9b970f6479a9a7d238bd3b4de92/package.json

"optionalDependencies": {
    "fsevents": "~2.1.2"
  },

Desired Behaviour

There are several ways this could be done. I think the least awful way to do this is the following:

...
 "platformDependencies": [
    {
      "os": ["darwin"],
      //"arch": [],
      "dependencySet": ["fsevents"]
    },
    {
      "os": ["win32"],
      //"arch": [],
      "dependencySet": ["win32-api", "windows-iana"]
    },
    {
      "os": ["darwin","linux","openbsd",...],
      "arch": ["x32","x64"],
      "dependencySet": ["posix"]
    },
    {
      "os": ["darwin","linux","openbsd",...],
      "arch": ["!x32","!x64"],
      "dependencySet": ["posix"]
    },
  ],
...

There would be a set of objects where a set of OSes and architectures can be specified along with the actual packages. At least 1 of the 2 of os and arch would be required but both being usable, each consisting of at least 1 target. As can be done in the current os and cpu values, the not! syntax can be used on platforms where not required.

References

  • n/a

Note: Currently, npm allows packages to specify that they're specific to an OS or architecture. This is a good feature that should be retained. This RFC is not meant to replace this feature but be used alongside it. optionalDependencies is also not being proposed for deprecation because it has use-cases beyond this poorly-fitting one.

You must be logged in to vote

Replies: 4 comments · 4 replies

Comment options

I think the problem identified here is a great one to discuss, I agree it is a problem we should try to solve. That said I am concerned the solution is very complicated when really what we want is not to show this kind of warning to the user at install time. Seems like we should also consider other solutions before considering this rather complicated one.

You must be logged in to vote
1 reply
@natewatson999
Comment options

I fully admit that this solution is too complicated. I wrote "least awful" instead of "best" because I couldn't think of any other scalable solution.

Comment options

There are two changes coming in npm v7 that make this less of a problem, and may make this feature less necessary.

The first is that build failures for optional dependencies are not displayed to the user. Install scripts are run in a child process and their output collected, and then only printed if the script exits with a non-zero error code.

Second, optional dependencies that specify an engine/os/cpu configuration that is not met will be ignored when building the tree. (Not fetched, not unpacked, etc.) So, fsevents will no longer be installed as a dependency of chokidar on any non-Darwin OS, without any warnings.

The tricky thing about platform-specific dependencies is that we may end up constructing a radically different tree on different platforms. Ie, not just "fsevents and its deps are not installed", but "linux gets foo@1 and darwin gets foo@2, in the same place in the tree". That makes it much harder to have build consistency, and leads to a lot of "works on my machine". (To some extent, optionalDependencies with platform declarations can lead to this as well, but it's a smaller surface.)

The other challenging thing about going down this road is that creating a declarative syntax for case statements inevitably leads to surprising amounts of complexity. (#129 is a great example of this!) Especially when we add negating, there's lots of opportunity for contradictions and confusing edge cases. That's not to say we can't or shouldn't do something like this, just that there's more design cost than it might appear at first. Before ratifying or implementing this feature, we'd have to do quite a bit of work to explore those edge cases as best as possible, or end up regretting it for years to come.

You must be logged in to vote
1 reply
@rjgotten
Comment options

The tricky thing about platform-specific dependencies is that we may end up constructing a radically different tree on different platforms. Ie, not just "fsevents and its deps are not installed", but "linux gets foo@1 and darwin gets foo@2, in the same place in the tree". That makes it much harder to have build consistency, and leads to a lot of "works on my machine". (To some extent, optionalDependencies with platform declarations can lead to this as well, but it's a smaller surface.)

As long as the platform identifiers are a known, quantifiable, and bounded set you can pre-create a set of platform-specific lock files for all necessary platforms. package-lock.json becomes package-lock.win.json; package-lock.linux.json etc. depending on which different platform identifiers actually occur within the package tree.

That way npm ci would still be able to just take whatever is in the lockfile for the target machine's platform, and be predictible and reproducible.

Optionally, you can have a feature that allows bounding the lock files that will be generated, by allowing the user to specify a set of --target-platforms=... This could be picked up automatically for all NPM operations within a project by putting it into an .npmrc file.

All of this would be statically analyzable and much more robust than the current skullduggery of e.g. having to use postinstall-scripts to wire up the correct platform-specific package implementations.

Comment options

Coming to this RFC after investigating options for an existing cli written in go to be published to npm seems like a nightmare. esbuild by evanw does an admirable job making that package work anywhere but if you look at the hoops gone through to get this working, the fact that npm doesn't natively support something like this seems like madness. Especially as we're seeing more and more tools written in rust/go/etc, this seems like this is only going to get worse.

You must be logged in to vote
2 replies
@paul-sachs
Comment options

Add to this that apparently on windows, if you have two optional dependencies with the same bin entries (to support platform-specific binaries) the resulting .bin directory is just empty. This might be a bug but it does raise the concern of using optional dependencies for these kinds of situations. What does it mean if two optional dependencies have overlapping binaries associated with them?

@alshdavid
Comment options

I have the same issue and have taken a relatively simple approach to solve it.

Main package that overrides the bin target with a symlink to the target binary via a postinstall script
https://github.com/alshdavid/mach/tree/main/npm/mach

OS specific packages (optionalDependencies)
https://github.com/alshdavid/mach/blob/main/npm/mach-os-arch/package.json

I tested this on Linux, MacOS and Windows and it works with npm, pnpm, yarn1 and yarn4

Comment options

Another solution that could be implemented at the package manager level is expanding binary targets to include platform discriminators - allowing them to work in combination with optionalDependencies .

Will also need some mechanism to forward the bin target to another package.

{
  "name": "@alshdavid/mach",
  "version": "0.0.25",
  "license": "MIT",
  "bin": {
    "mach": "bin/mach"
  },
  "binMeta": {
    "mach": [
      { "os": ["win32"], "arch": ["x64"], "forward": "@alshdavid/mach-windows-amd64#bin.mach" },
      { "os": ["linux"], "arch": ["x64"], "forward": "@alshdavid/mach-linux-amd64#bin.mach" },
      { "os": ["darwin"], "arch": ["x64"], "forward": "@alshdavid/mach-macos-amd64#bin.mach" },
      // etc
    ]
  },
  "optionalDependencies": {
    "@alshdavid/mach-linux-amd64": "0.0.25",
    "@alshdavid/mach-linux-arm64": "0.0.25",
    "@alshdavid/mach-macos-amd64": "0.0.25",
    "@alshdavid/mach-macos-arm64": "0.0.25",
    "@alshdavid/mach-windows-amd64": "0.0.25",
    "@alshdavid/mach-windows-arm64": "0.0.25"
  }
}

optionalDependencies will install packages that match their internal os and arch fields and the main package will select the bin to link based on the relevant target.

You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
6 participants
Converted from issue

This discussion was converted from issue #120 on May 08, 2020 14:33.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.