Tasks
When building a package, you often have to do more than just run the code. Steps like formatting, linting, compiling, testing, benchmarking, etc. are often part of a project. With pixi tasks, this should become much easier to do.
Here are some quick examples
[tasks]
# Commands as lists so you can also add documentation in between.
configure = { cmd = [
"cmake",
# Use the cross-platform Ninja generator
"-G",
"Ninja",
# The source is in the root directory
"-S",
".",
# We wanna build in the .build directory
"-B",
".build",
] }
# Depend on other tasks
build = { cmd = ["ninja", "-C", ".build"], depends-on = ["configure"] }
# Using environment variables
run = "python main.py $PIXI_PROJECT_ROOT"
set = "export VAR=hello && echo $VAR"
# Cross platform file operations
copy = "cp pixi.toml pixi_backup.toml"
clean = "rm pixi_backup.toml"
move = "mv pixi.toml backup.toml"
Depends on#
Just like packages can depend on other packages, our tasks can depend on other tasks. This allows for complete pipelines to be run with a single command.
An obvious example is compiling before running an application.
Checkout our cpp_sdl
example for a running example.
In that package we have some tasks that depend on each other, so we can assure that when you run pixi run start
everything is set up as expected.
pixi task add configure "cmake -G Ninja -S . -B .build"
pixi task add build "ninja -C .build" --depends-on configure
pixi task add start ".build/bin/sdl_example" --depends-on build
Results in the following lines added to the pixi.toml
[tasks]
# Configures CMake
configure = "cmake -G Ninja -S . -B .build"
# Build the executable but make sure CMake is configured first.
build = { cmd = "ninja -C .build", depends-on = ["configure"] }
# Start the built executable
start = { cmd = ".build/bin/sdl_example", depends-on = ["build"] }
The tasks will be executed after each other:
- First
configure
because it has no dependencies. - Then
build
as it only depends onconfigure
. - Then
start
as all it dependencies are run.
If one of the commands fails (exit with non-zero code.) it will stop and the next one will not be started.
With this logic, you can also create aliases as you don't have to specify any command in a task.
Results in the following pixi.toml
.
Now run both tools with one command.
Working directory#
Pixi tasks support the definition of a working directory.
cwd
" stands for Current Working Directory.
The directory is relative to the pixi package root, where the pixi.toml
file is located.
Consider a pixi project structured as follows:
To add a task to run the bar.py
file, use:
This will add the following line to manifest file:
Caching#
When you specify inputs
and/or outputs
to a task, pixi will reuse the result of the task.
For the cache, pixi checks that the following are true:
- No package in the environment has changed.
- The selected inputs and outputs are the same as the last time the task was run. We compute fingerprints of all the files selected by the globs and compare them to the last time the task was run.
- The command is the same as the last time the task was run.
If all of these conditions are met, pixi will not run the task again and instead use the existing result.
Inputs and outputs can be specified as globs, which will be expanded to all matching files.
[tasks]
# This task will only run if the `main.py` file has changed.
run = { cmd = "python main.py", inputs = ["main.py"] }
# This task will remember the result of the `curl` command and not run it again if the file `data.csv` already exists.
download_data = { cmd = "curl -o data.csv https://example.com/data.csv", outputs = ["data.csv"] }
# This task will only run if the `src` directory has changed and will remember the result of the `make` command.
build = { cmd = "make", inputs = ["src/*.cpp", "include/*.hpp"], outputs = ["build/app.exe"] }
Note: if you want to debug the globs you can use the --verbose
flag to see which files are selected.
Environment variables#
You can set environment variables for a task. These are seen as "default" values for the variables as you can overwrite them from the shell.
If you runpixi run echo
it will output hello
.
When you set the environment variable ARGUMENT
before running the task, it will use that value instead.
These variables are not shared over tasks, so you need to define these for every task you want to use them in.
Extend instead of overwrite
If you use the same environment variable in the value as in the key of the map you will also overwrite the variable.
For example overwriting a PATH
/tmp/path:/usr/bin:/bin
instead of the original /usr/bin:/bin
.
Clean environment#
You can make sure the environment of a task is "pixi only".
Here pixi will only include the minimal required environment variables for your platform to run the command in.
The environment will contain all variables set by the conda environment like "CONDA_PREFIX"
.
It will however include some default values from the shell, like:
"DISPLAY"
, "LC_ALL"
, "LC_TIME"
, "LC_NUMERIC"
, "LC_MEASUREMENT"
, "SHELL"
, "USER"
, "USERNAME"
, "LOGNAME"
, "HOME"
, "HOSTNAME"
,"TMPDIR"
, "XPC_SERVICE_NAME"
, "XPC_FLAGS"
pixi run --clean-env TASK_NAME
.
clean-env
not supported on Windows
On Windows it's hard to create a "clean environment" as conda-forge
doesn't ship Windows compilers and Windows needs a lot of base variables.
Making this feature not worthy of implementing as the amount of edge cases will make it unusable.
Our task runner: deno_task_shell#
To support the different OS's (Windows, OSX and Linux), pixi integrates a shell that can run on all of them.
This is deno_task_shell
.
The task shell is a limited implementation of a bourne-shell interface.
Built-in commands#
Next to running actual executable like ./myprogram
, cmake
or python
the shell has some built-in commandos.
cp
: Copies files.mv
: Moves files.rm
: Remove files or directories. Ex:rm -rf [FILE]...
- Commonly used to recursively delete files or directories.mkdir
: Makes directories. Ex.mkdir -p DIRECTORY...
- Commonly used to make a directory and all its parents with no error if it exists.pwd
: Prints the name of the current/working directory.sleep
: Delays for a specified amount of time. Ex.sleep 1
to sleep for 1 second,sleep 0.5
to sleep for half a second, orsleep 1m
to sleep a minuteecho
: Displays a line of text.cat
: Concatenates files and outputs them on stdout. When no arguments are provided, it reads and outputs stdin.exit
: Causes the shell to exit.unset
: Unsets environment variables.xargs
: Builds arguments from stdin and executes a command.
Syntax#
- Boolean list: use
&&
or||
to separate two commands.&&
: if the command before&&
succeeds continue with the next command.||
: if the command before||
fails continue with the next command.
- Sequential lists: use
;
to run two commands without checking if the first command failed or succeeded. - Environment variables:
- Set env variable using:
export ENV_VAR=value
- Use env variable using:
$ENV_VAR
- unset env variable using
unset ENV_VAR
- Set env variable using:
- Shell variables: Shell variables are similar to environment variables, but won’t be exported to spawned commands.
- Set them:
VAR=value
- use them:
VAR=value && echo $VAR
- Set them:
- Pipelines: Use the stdout output of a command into the stdin a following command
|
:echo Hello | python receiving_app.py
|&
: use this to also get the stderr as input.
- Command substitution:
$()
to use the output of a command as input for another command.python main.py $(git rev-parse HEAD)
- Negate exit code:
!
before any command will negate the exit code from 1 to 0 or visa-versa. - Redirects:
>
to redirect the stdout to a file.echo hello > file.txt
will puthello
infile.txt
and overwrite existing text.python main.py 2> file.txt
will put thestderr
output infile.txt
.python main.py &> file.txt
will put thestderr
andstdout
infile.txt
.echo hello > file.txt
will appendhello
to the existingfile.txt
.
- Glob expansion:
*
to expand all options.echo *.py
will echo all filenames that end with.py
echo **/*.py
will echo all filenames that end with.py
in this directory and all descendant directories.echo data[0-9].csv
will echo all filenames that have a single number afterdata
and before.csv
More info in deno_task_shell
documentation.