Use S3 with pixi#
If you want to use S3 object storage to fetch your packages, you can use the s3:// protocol as a channel.
In the bucket, your objects need to adhere to the standard conda repository structure:
Pixi supports two ways to configure access to your S3 bucket:
- Using AWS credentials from environment variables or AWS configuration files, like in any other AWS tool
- Using pixi's configuration in combination with storing the credentials in pixi's authentication storage
These two options are mutually exclusive!
Specifying s3-options (see below) will deactivate the AWS credentials fetching.
You can either use the AWS credentials from the conventional locations (by not specifying s3-options) or from pixi's authentication storage (by specifying s3-options).
Using AWS configuration#
You can specify AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in your environment variables for pixi to use them.
You can also specify AWS_CONFIG_FILE and AWS_PROFILE to use a custom AWS configuration file and profile.
[profile conda]
sso_account_id = 123456789012
sso_role_name = PowerUserAccess
sso_start_url = https://my-company.awsapps.com/start
region = eu-central-1
output = json
$ export AWS_CONFIG_FILE=/path/to/aws.config
$ export AWS_PROFILE=conda
$ aws sso login
Attempting to automatically open the SSO authorization page in your default browser.
If the browser does not open or you wish to use a different device to authorize this request, open the following URL:
https://my-company.awsapps.com/start/#/device
Then enter the code:
DTBC-WFXC
Successfully logged into Start URL: https://my-company.awsapps.com/start
$ pixi search -c s3://my-s3-bucket/channel my-private-package
# ...
jobs:
ci:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
# temporary credentials via OIDC
- name: Log in to AWS
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-poweruser
aws-region: eu-central-1
- name: Set up pixi
# AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set by aws-actions/configure-aws-credentials
uses: prefix-dev/setup-pixi@v0.8.3
Using pixi's configuration#
You can specify the project.s3-options in your pixi.toml file.
This might be useful when you want to use a custom S3-compatible host and not AWS's configuration.
[project.s3-options.my-bucket]
endpoint-url = "https://my-s3-host"
region = "us-east-1"
force-path-style = false
You need to configure this per bucket you use, i.e. use [project.s3-options.<bucket-name>].
$ pixi auth login --aws-access-key-id=... --aws-secret-access-key=... s3://my-s3-bucket
Authenticating with s3://my-s3-bucket
$ pixi search my-private-package
# ...
You can also specify the s3-options in your pixi configuration.
[s3-options.my-bucket]
endpoint-url = "https://my-s3-host"
region = "us-east-1"
force-path-style = false
jobs:
ci:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
# temporary credentials via OIDC
- name: Log in to AWS
uses: aws-actions/configure-aws-credentials@v4
id: aws
with:
role-to-assume: arn:aws:iam::123456789012:role/github-poweruser
aws-region: eu-central-1
- name: Set up pixi
# AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set by aws-actions/configure-aws-credentials
uses: prefix-dev/setup-pixi@v0.8.3
with:
auth-s3-access-key-id: ${{ steps.aws.outputs.aws-access-key-id }}
auth-s3-secret-access-key: ${{ steps.aws.outputs.aws-secret-access-key }}
auth-s3-session-token: ${{ steps.aws.outputs.aws-session-token }}
auth-host: s3://my-s3-bucket
Public S3 buckets#
Public buckets that don't need authentication can be used by just specifying the endpoint as a regular https URL.
For example, on AWS, you might have a bucket that is publicly accessible via https://my-public-bucket.s3.eu-central-1.amazonaws.com.
Note that for this, you need to configure your S3 bucket in such a way that it allows public access.
On AWS, you need the GetObject and ListBucket permissions for this.
Here is an example policy for AWS S3:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadGetObject",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::my-public-bucket/*"
},
{
"Sid": "PublicReadListBucket",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::my-public-bucket"
}
]
}
Cloudflare R2 also supports public buckets through a Cloudflare-managed r2.dev subdomain or a custom domain under your control, see here.
S3-compatible storage#
Many other cloud providers offer S3-compatible storage APIs.
You can use them with pixi by specifying the s3-options in your manifest file.
MinIO#
[project.s3-options]
endpoint-url = "https://minio.example.com"
region = "us-east-1"
force-path-style = true
Cloudflare R2#
endpoint-url = "https://<account-id>.eu.r2.cloudflarestorage.com"
region = "WEUR"
force-path-style = false
Wasabi#
Backblaze B2#
endpoint-url = "https://s3.us-west-004.backblazeb2.com"
region = "us-west-004"
force-path-style = true
Google Cloud Storage#
Note pixi also supports gcs:// URLs.
endpoint-url = "https://storage.googleapis.com"
region = "us-east-1"
force-path-style = false
Hetzner Object Storage#
endpoint-url = "https://fsn1.your-objectstorage.com"
region = "US"
force-path-style = false
Uploading to S3#
You can upload to S3 using rattler-build upload s3.
For more information, see rattler-build's documentation.