@mapbox/node-pre-gyp

@mapbox/node-pre-gyp makes it easy to publish and install Node.js C++ addons from binaries

@mapbox/node-pre-gyp stands between npm and node-gyp and offers a cross-platform method of binary deployment.

Special note on previous package

On Feb 9th, 2021 @mapbox/node-pre-gyp@1.0.0 was released. Older, unscoped versions that are not part of the @mapbox org are deprecated and only @mapbox/node-pre-gyp will see updates going forward. To upgrade to the new package do:

npm uninstall node-pre-gyp --save
npm install @mapbox/node-pre-gyp --save

Features

• A command line tool called node-pre-gyp that can install your package's C++ module from a binary.
• A variety of developer targeted commands for packaging, testing, and publishing binaries.
• A JavaScript module that can dynamically require your installed binary: require('@mapbox/node-pre-gyp').find

For a hello world example of a module packaged with node-pre-gyp see https://github.com/springmeyer/node-addon-example and the wiki for real world examples.

Credits

• The module is modeled after node-gyp by @Tootallnate
• Motivation for initial development came from @ErisDS and the Ghost Project.
• Development is sponsored by Mapbox

FAQ

See the Frequently Ask Questions.

Depends

We will attempt to track the Node.js release schedule and will regularly retire support for versions that have reached EOL.

• v2: Node.js >= 18.x (unreleased)
• v1: Node.js >= 8.x

Install

node-pre-gyp is designed to be installed as a local dependency of your Node.js C++ addon and accessed like:

./node_modules/.bin/node-pre-gyp --help

But you can also install it globally:

npm install @mapbox/node-pre-gyp -g

Usage

Commands

View all possible commands:

node-pre-gyp --help

• clean - Remove the entire folder containing the compiled .node module
• install - Install pre-built binary for module
• reinstall - Run "clean" and "install" at once
• build - Compile the module by dispatching to node-gyp or nw-gyp
• rebuild - Run "clean" and "build" at once
• package - Pack binary into tarball
• testpackage - Test that the staged package is valid
• publish - Publish pre-built binary
• unpublish - Unpublish pre-built binary
• info - Fetch info on published binaries

You can also chain commands:

node-pre-gyp clean build unpublish publish info

Options

Options include:

• -C/--directory: run the command in this directory
• --build-from-source: build from source instead of using pre-built binary
• --update-binary: reinstall by replacing previously installed local binary with remote binary
• --runtime=node-webkit: customize the runtime: node, electron and node-webkit are the valid options
• --fallback-to-build: fallback to building from source if pre-built binary is not available
• --target=0.4.0: Pass the target node or node-webkit version to compile against
• --target_arch=ia32: Pass the target arch and override the host arch. Any value that is supported by Node.js is valid.
• --target_platform=win32: Pass the target platform and override the host platform. Valid values are linux, darwin, win32, sunos, freebsd, openbsd, and aix.
• --acl=<acl>: Set the S3 ACL when publishing binaries (e.g., public-read, private). Overrides the binary.acl setting in package.json.

Both --build-from-source and --fallback-to-build can be passed alone or they can provide values. You can pass --fallback-to-build=false to override the option as declared in package.json. In addition to being able to pass --build-from-source you can also pass --build-from-source=myapp where myapp is the name of your module.

For example: npm install --build-from-source=myapp. This is useful if:

• myapp is referenced in the package.json of a larger app and therefore myapp is being installed as a dependency with npm install.
• The larger app also depends on other modules installed with node-pre-gyp
• You only want to trigger a source compile for myapp and the other modules.

Configuring

This is a guide to configuring your module to use node-pre-gyp.

1) Add new entries to your package.json

• Add @mapbox/node-pre-gyp to dependencies
• Add @aws-sdk/client-s3 as a devDependency
• Add a custom install script
• Declare a binary object

This looks like:

    "dependencies"  : {
      "@mapbox/node-pre-gyp": "1.x"
    },
    "devDependencies": {
      "@aws-sdk/client-s3": "3.x"
    }
    "scripts": {
        "install": "node-pre-gyp install --fallback-to-build"
    },
    "binary": {
        "module_name": "your_module",
        "module_path": "./lib/binding/",
        "host": "https://your_module.s3-us-west-1.amazonaws.com"
    }

For a full example see node-addon-examples's package.json.

Let's break this down:

• Dependencies need to list node-pre-gyp
• Your devDependencies should list @aws-sdk/client-s3 so that you can run node-pre-gyp publish locally or a CI system. We recommend using devDependencies only since @aws-sdk/client-s3 is large and not needed for node-pre-gyp install since it only uses http to fetch binaries
• Your scripts section should override the install target with "install": "node-pre-gyp install --fallback-to-build". This allows node-pre-gyp to be used instead of the default npm behavior of always source compiling with node-gyp directly.
• Your package.json should contain a binary section describing key properties you provide to allow node-pre-gyp to package optimally. They are detailed below.

Note: in the past we recommended putting @mapbox/node-pre-gyp in the bundledDependencies, but we no longer recommend this. In the past there were npm bugs (with node versions 0.10.x) that could lead to node-pre-gyp not being available at the right time during install (unless we bundled). This should no longer be the case. Also, for a time we recommended using "preinstall": "npm install @mapbox/node-pre-gyp" as an alternative method to avoid needing to bundle. But this did not behave predictably across all npm versions - see https://github.com/mapbox/node-pre-gyp/issues/260 for the details. So we do not recommend using preinstall to install @mapbox/node-pre-gyp. More history on this at https://github.com/strongloop/fsevents/issues/157#issuecomment-265545908.

The binary object has three required properties

module_name

The name of your native node module. This value must:

• Match the name passed to the NODE_MODULE macro
• Must be a valid C variable name (e.g. it cannot contain -)
• Should not include the .node extension.

module_path

The location your native module is placed after a build. This should be an empty directory without other Javascript files. This entire directory will be packaged in the binary tarball. When installing from a remote package this directory will be overwritten with the contents of the tarball.

Note: This property supports variables based on Versioning.

host

A url to the remote location where you've published tarball binaries (must be https not http).

It is highly recommended that you use Amazon S3. The reasons are:

• Various node-pre-gyp commands like publish and info only work with an S3 host.
• S3 is a very solid hosting platform for distributing large files.
• We provide detail documentation for using S3 hosting with node-pre-gyp.

Why then not require S3? Because while some applications using node-pre-gyp need to distribute binaries as large as 20-30 MB, others might have very small binaries and might wish to store them in a GitHub repo. This is not recommended, but if an author really wants to host in a non-S3 location then it should be possible.

It should also be mentioned that there is an optional and entirely separate npm module called node-pre-gyp-github which is intended to complement node-pre-gyp and be installed along with it. It provides the ability to store and publish your binaries within your repositories GitHub Releases if you would rather not use S3 directly. Installation and usage instructions can be found here, but the basic premise is that instead of using the node-pre-gyp publish command you would use node-pre-gyp-github publish.

The binary object other optional S3 properties

If you are not using a standard s3 path like bucket_name.s3(.-)region.amazonaws.com, you might get an error on publish because node-pre-gyp extracts the region and bucket from the host url. For example, you may have an on-premises s3-compatible storage server, or may have configured a specific dns redirecting to an s3 endpoint. In these cases, you can explicitly set the region and bucket properties to tell node-pre-gyp to use these values instead of guessing from the host property. The following values can be used in the binary section:

host

The url to the remote server root location (must be https not http).

bucket

The bucket name where your tarball binaries should be located.

region

Your S3 server region.

s3ForcePathStyle

Set s3ForcePathStyle to true if the endpoint url should not be prefixed with the bucket name. If false (default), the server endpoint would be constructed as bucket_name.your_server.com.

acl

The S3 Access Control List (ACL) to apply when publishing binaries. Defaults to 'public-read' for backward compatibility. Common values include:

• public-read - (default) Binary is publicly accessible by anyone
• private - Binary requires AWS credentials to download
• authenticated-read - Any authenticated AWS user can download
• bucket-owner-read - Bucket owner gets READ access
• bucket-owner-full-control - Bucket owner gets FULL_CONTROL access

For private binaries:

• Users installing your package will need AWS credentials configured (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables)
• The @aws-sdk/client-s3 package must be available at install time
• If authentication fails, node-pre-gyp will fall back to building from source (if --fallback-to-build is specified)

You can also specify the ACL via command-line flag: node-pre-gyp publish --acl=private

Example for private binaries:

"binary": {
  "module_name": "your_module",
  "module_path": "./lib/binding/",
  "host": "https://your-bucket.s3.us-east-1.amazonaws.com",
  "acl": "private"
}

The binary object has optional properties

remote_path

It is recommended that you customize this property. This is an extra path to use for publishing and finding remote tarballs. The default value for remote_path is "" meaning that if you do not provide it then all packages will be published at the base of the host. It is recommended to provide a value like ./{name}/v{version} to help organize remote packages in the case that you choose to publish multiple node addons to the same host.

Note: This property supports variables based on Versioning.

package_name

It is not recommended to override this property unless you are also overriding the remote_path. This is the versioned name of the remote tarball containing the binary .node module and any supporting files you've placed inside the module_path directory. Unless you specify package_name in your package.json then it defaults to {module_name}-v{version}-{node_abi}-{platform}-{arch}.tar.gz which allows your binary to work across node versions, platforms, and architectures. If you are using remote_path that is also versioned by ./{module_name}/v{version} then you could remove these variables from the package_name and just use: {node_abi}-{platform}-{arch}.tar.gz. Then your remote tarball will be looked up at, for example, https://example.com/your-module/v0.1.0/node-v11-linux-x64.tar.gz.

Avoiding the version of your module in the package_name and instead only embedding in a directory name can be useful when you want to make a quick tag of your module that does not change any C++ code. In this case you can just copy binaries to the new version behind the scenes like:

aws s3 sync --acl public-read s3://mapbox-node-binary/sqlite3/v3.0.3/ s3://mapbox-node-binary/sqlite3/v3.0.4/

Note: This property supports variables based on Versioning.

2) Add a new target to binding.gyp

node-pre-gyp calls out to node-gyp to compile the module and passes variables along like module_name and module_path.

A new target must be added to binding.gyp that moves the compiled .node module from ./build/Release/module_name.node into the directory specified by module_path.

Add a target like this at the end of your targets list:

    {
      "target_name": "action_after_build",
      "type": "none",
      "dependencies": [ "<(module_name)" ],
      "copies": [
        {
          "files": [ "<(PRODUCT_DIR)/<(module_name).node" ],
          "destination": "<(module_path)"
        }
      ]
    }

For a full example see node-addon-example's binding.gyp.

3) Dynamically require your .node

Inside the main js file that requires your addon module you are likely currently doing:

var binding = require('../build/Release/binding.node');

or:

var bindings = require('./bindings')

Change those lines to:

var binary = require('@mapbox/node-pre-gyp');
var path = require('path');
var binding_path = binary.find(path.resolve(path.join(__dirname,'./package.json')));
var binding = require(binding_path);

For a full example see [node-addon-example's index.js](https://github.com/springmeyer/node-addon-example/blob/2ff60a8ded7f042864ad21db00c3a5a06c