Reference
Recipe structure
A recipe file with all available fields is shown below. Each field is described afterward.
minimum_required_version: 0.4.0
env:
OUTPUT_DIR: ~/output
recipe:
first_step:
tag:
- first
description: >
Description of the first step
env:
USER_NAME: !input
name: MY_NAME
default: Juhani
required: false
cwd: $OUTPUT_DIR
run:
- echo 'Hello from recp'
- cmd: echo "My name is $USER_NAME and today is {DAY}"
apply:
- fn: date
args:
token: "{DAY}"
format: "%A"
.yaml file where:
minimum_required_versionis an optional field specifying the minimumrecpversion required to run it.envis an optional field containing environment variables shared by all steps. A step can override them in its ownenvkey.recipeis the required root key containing the description of each step to be run.
Each step has:
- A required name key (
first_step) containing the information relative to that particular step. - An optional
tagkey containing astror alistofstr, each one corresponding to a tag that can be used to select a certain subset of steps to be run with the--tagoption. - An optional
descriptionkey containing astrthat describes what the step does. This is only for documentation purposes, and is printed to the terminal when the recipe is run. - An optional
envkey containing environment variables to be used in that step. They are expanded in the commands and also passed to the processes run by the commands. The environment variables of a specific step are isolated from all other steps. To share variables across steps you can use the recipe-levelenvkey. Additionally, environment variables can be defined directly in the.yamlfile, or defined by the user when running the recipe. User defined variables can be configured using the!inputconstructor that has a mandatorynamekey, and an optionaldefaultkey to define the default value, and an optionalrequiredkey containing aboolvalue to specify whether that variable is required or optional. - An optional
cwdkey containing the folder where the commands of that step are run. - An optional
parallelkey containing the number of commands of that step run at the same time (see Parallel commands). - A required
runkey that contains astror alistof the commands to be run. A command can be astror adictcontaining thecmdkey and theapplykey containing alistof modifiers to be applied to that command as a pre-processing step to generate one or multiple commands. Each modifier is defined by afnkey with the modifier function name, and anargskey with the corresponding arguments. Adictcommand can also contain askip_if_existskey (see Skipping existing outputs) and alabelkey (see Command labels).
Environment variables such as $HOME or $USER_NAME are expanded by recp before the commands are run. Use $$ to pass a literal $ to the command instead. The RECP_RECIPE_FILE variable contains the absolute path to the recipe file being run.
A recipe can be run using:
Additionally, you can show the commands without running them with the --dry-run option, that can be used for debugging purposes.
To filter steps by tag, you can use:
This will only collect steps containing the tag first. Multiple tags can be used to filter a step, in which case only steps containing all of them will be selected.
To run only some steps by name, or to resume a recipe from a given step, you can use:
recp run /path/to/your/recipe.yaml --step first_step
recp run /path/to/your/recipe.yaml --from first_step
If a command fails, recp stops, prints a summary and exits with the exit code of the failed command. With the --ignore-errors option, the remaining commands are run anyway and recp exits with code 1 if any of them failed.
The output is colored only when printed to a terminal. Colors can be disabled by setting the NO_COLOR environment variable, or forced by setting FORCE_COLOR.
To see the steps of a recipe without running it or prompting for any value, you can use:
For further options on recp run, you can run:
Constructors
A recipe .yaml file can be created using a certain set of custom constructors to facilitate expressing your commands.
!input constructor
The !input constructor allows you to introduce user variables provided through the terminal into your recipe. It is typically used in the env key as exemplified below.
minimum_required_version: 0.4.0
recipe:
step:
env:
USER_VAR: !input
name: USER_VAR
default: default_value
required: false
run:
- "echo 'print user variable: $USER_VAR'"
In this case:
nameis a required key with the name of the environment variable to be provided by the user. This name can be the same as its parent key or different.defaultis an optional key. It sets the value to use if the user does not provide the variable.requiredis an optional key. Iftrue, the user must provide the variable to run the recipe.
The recipe can be run as:
It will result in the following output:
Using recipe '/path/to/input_constructor_example.yaml' ...
1 step(s) found
Pre-processing recipe data ...
Recipe pre-processing successfully completed
Parsing step environments ...
Step environments processing successfully completed
Pre-processing recipe commands ...
Recipe commands successfully completed
Running commands ...
[1/1] Running step 'step' ...
Command: echo 'print user variable: Juhani'
print user variable: Juhani
Summary: 1 step(s) completed, 0 failed, 0 not run
!prompt constructor
Info
This constructor was introduced in version 0.1.2, so update your minimum_required_version if that key is present.
The !prompt constructor works like !input, but instead of directly including the variable in your command, it creates a user prompt for entering a value or choice. It's usually used within the env key, as shown below.
minimum_required_version: 0.4.0
recipe:
step:
env:
USER_VAR: !prompt
message: Introduce variable value
run:
- "echo 'print user variable: $USER_VAR'"
In this case message is a required key containing the message displayed to the user. You can run the recipe with:
The result will depend on what the user typed before pressing enter.
1 step(s) found
Pre-processing recipe data ...
Recipe pre-processing successfully completed
Parsing step environments ...
Introduce variable value: Juhani
Step environments processing successfully completed
Pre-processing recipe commands ...
Recipe commands successfully completed
Running commands ...
[1/1] Running step 'step' ...
Command: echo 'print user variable: Juhani'
print user variable: Juhani
Summary: 1 step(s) completed, 0 failed, 0 not run
Additionally, the input can be configured to be a set of choices:
minimum_required_version: 0.4.0
recipe:
step:
env:
CHOICE: !prompt
message: Favourite pet
choices:
- cat
- dog
run:
- "echo 'You are a $CHOICE person'"
In this case, the user can only type one of the available choices. If they enter something else, the program will prompt them to select a valid choice again. For example, if you type cat and press Enter, the output is:
1 step(s) found
Pre-processing recipe data ...
Recipe pre-processing successfully completed
Parsing step environments ...
Favourite pet [cat/dog]: cat
Step environments processing successfully completed
Pre-processing recipe commands ...
Recipe commands successfully completed
Running commands ...
[1/1] Running step 'step' ...
Command: echo 'You are a cat person'
You are a cat person
Summary: 1 step(s) completed, 0 failed, 0 not run
Additionally, you can set a default choice using the default key:
minimum_required_version: 0.4.0
recipe:
step:
env:
CHOICE: !prompt
message: Favourite pet
choices:
- cat
- dog
default: cat
run:
- "echo 'You are a $CHOICE person'"
In this case, the option prefixed with a * in the prompt will be selected if the user presses enter without explicitly choosing another option.
Leading and trailing spaces are removed from the user response. If no input is available (e.g. when running in a batch job), the default value is used if there is one, otherwise recp stops with an error.
!split constructor
The !split constructor splits a str by whitespace into a list of str. It is useful to pass multiple values to a modifier.
!lines constructor
Info
This constructor was introduced in version 0.3.0, so update your minimum_required_version if that key is present.
The !lines constructor reads a text file and returns a list with one item per line. Leading and trailing spaces are removed, and empty lines and lines starting with # are ignored. Relative paths are resolved from the folder where recp is run.
minimum_required_version: 0.4.0
recipe:
step:
run:
- cmd: echo 'Processing speaker __SPEAKER__'
apply:
- fn: replace
args:
__SPEAKER__: !lines speakers.txt
!shell constructor
Info
This constructor was introduced in version 0.3.0, so update your minimum_required_version if that key is present.
The !shell constructor runs a command when the recipe is loaded and returns its output as a str. The command is run even when using --dry-run.
minimum_required_version: 0.4.0
recipe:
step:
env:
COMMIT: !shell git rev-parse --short HEAD
run:
- echo 'Current commit is $COMMIT'
!now constructor
Info
This constructor was introduced in version 0.3.0, so update your minimum_required_version if that key is present.
The !now constructor returns the date and time when the recipe was loaded, using a datetime format (%Y-%m-%d by default). All !now constructors in a recipe use the same timestamp, so the value is consistent across steps.
minimum_required_version: 0.4.0
env:
RUN_ID: !now "%Y%m%d-%H%M"
recipe:
step:
run:
- echo 'Saving results to results/$RUN_ID'
!expr constructor
The !expr constructor allows using python expressions to be evaluated at runtime.
Warning
This constructor uses eval() internally and is considered unsafe. By default, it is disabled, but you can enable it by using the --unsafe option when running the recipe.
This constructor can be used in any part of your recipe as follows:
minimum_required_version: 0.4.0
recipe:
step:
env:
RESULT: !expr 24 * 60 * 60
run:
- echo 'A day has $RESULT seconds'
To run this recipe and enable the !expr constructor, you must use the --unsafe option
as follows:
It will result in the following output:
Using recipe '/path/to/expr_constructor_example.yaml' ...
1 step(s) found
Pre-processing recipe data ...
Recipe pre-processing successfully completed
Parsing step environments ...
Step environments processing successfully completed
Pre-processing recipe commands ...
Recipe commands successfully completed
Running commands ...
[1/1] Running step 'step' ...
Command: echo 'A day has 86400 seconds'
A day has 86400 seconds
Summary: 1 step(s) completed, 0 failed, 0 not run
Modifiers
Modifiers are functions that take a command as input and transform it into one or more commands according to some logic. They can be used to write more compact recipes and to add variation or dynamic behavior to your commands.
To use a modifier, you can include then in the run key as follows:
minimum_required_version: 0.4.0
recipe:
step:
run:
- cmd: echo 'This message will be repeated multiple times'
apply:
- fn: repeat
args:
n: 3
- The command in the
runkey is now adict. - The
applykey contains all modifiers to be applied. It is alistwhere each item is adictwith afnkey (the modifier function) and anargskey (the arguments passed to the modifier). - When multiple modifiers are defined, they are applied sequentially: the output of one modifier becomes the input to the next.
In this example, the repeat modifier will be applied to the command, causing it to be run multiple times, resulting in:
Using recipe '/path/to/modifier_example.yaml' ...
1 step(s) found
Pre-processing recipe data ...
Recipe pre-processing successfully completed
Parsing step environments ...
Step environments processing successfully completed
Pre-processing recipe commands ...
Recipe commands successfully completed
Running commands ...
[1/1] Running step 'step' ...
Command: echo 'This message will be repeated multiple times'
This message will be repeated multiple times
Command: echo 'This message will be repeated multiple times'
This message will be repeated multiple times
Command: echo 'This message will be repeated multiple times'
This message will be repeated multiple times
Summary: 1 step(s) completed, 0 failed, 0 not run
Info
The commands are unwrapped before being run, thus, you can use --dry-run the preview the commands that will be run after all modifiers are applied.
A list of supported modifiers is provided below. You can also implement your own by adding it to the src/recp/utils/apply.py file.
basename
Replaces a token by the basename of a given path.
| Name | Type | Description |
|---|---|---|
token |
str |
Token to be replaced. |
path |
str |
Path whose basename will be extracted. |
Example:
recipe:
step:
run:
- cmd: echo 'The file name is __NAME__'
apply:
- fn: basename
args:
token: __NAME__
path: /path/to/file.txt
Result:
cartesian_product
Generates one command per combination of the values of all tokens.
| Name | Type | Description |
|---|---|---|
**kwargs |
List[str] |
Keyword arguments where each key is a token and each value is a list of values. |
Example:
recipe:
step:
run:
- cmd: echo 'model __MODEL__ with seed __SEED__'
apply:
- fn: cartesian_product
args:
__MODEL__: [small, large]
__SEED__: [0, 1]
Result:
date
Replaces a token by a date in a specific format.
| Name | Type | Description | Default |
|---|---|---|---|
token |
str |
Token to be replaced. | |
format |
str |
Date format using datetime convention. |
%Y-%m-%d |
Example:
recipe:
step:
run:
- cmd: echo 'Today is __DATE__'
apply:
- fn: date
args:
token: __DATE__
format: "%Y-%m-%d"
Result:
dir_files
Expands a command creating one copy per file in a given folder.
| Name | Type | Description | Default |
|---|---|---|---|
token |
str |
Token to be replaced. | |
dir |
str |
Input folder. | |
ext |
str | List[str] |
Extension(s) to filter (e.g. wav or .wav, * includes all extensions). |
* |
recursive |
bool |
If True, dir is searched recursively. |
True |
name_token |
str |
Token replaced by the file name (e.g. file.wav). |
|
stem_token |
str |
Token replaced by the file name without extension (e.g. file). |
|
rel_token |
str |
Token replaced by the path relative to dir (e.g. sub/file.wav). |
|
rel_dir_token |
str |
Token replaced by the folder relative to dir (e.g. sub, or . for files directly in dir). |
Example:
recipe:
step:
env:
DIR: &dir /path/to/my/dir
run:
- cmd: echo '$DIR contains file __FILE__'
apply:
- fn: dir_files
args:
token: __FILE__
dir: *dir
Result:
/path/to/my/dir contains file /path/to/my/dir/00_file.yaml
/path/to/my/dir contains file /path/to/my/dir/01_file.yaml
...
The optional tokens make it possible to build an output path from each input file. For example, to convert all .wav files in a folder to .flac, keeping the same subfolders:
recipe:
step:
run:
- cmd: mkdir -p out/__REL_DIR__ && ffmpeg -i __FILE__ out/__REL_DIR__/__STEM__.flac
apply:
- fn: dir_files
args:
token: __FILE__
dir: corpus
ext: wav
stem_token: __STEM__
rel_dir_token: __REL_DIR__
index
Replaces a token by the command index value.
| Name | Type | Description | Default |
|---|---|---|---|
token |
str |
Token to replace. | |
offset |
int |
Offset applied to all values. | 0 |
zfill |
int |
Minimum width of the number, padded with leading zeros if needed. | 0 |
Example:
recipe:
step:
run:
- cmd: echo 'Command number __INDEX__'
apply:
- fn: repeat
args:
n: 3
- fn: index
args:
token: __INDEX__
zfill: 2
offset: 1
Result:
match
Replaces a token value based on a matching value of a given variable.
| Name | Type | Description |
|---|---|---|
var |
str |
Variable to match. |
token |
str |
Token to be replaced. |
choices |
List[str] |
List of choices. |
values |
List[str] |
List of values (correlative to choices). |
Example:
recipe:
step:
env:
CHOICE: &choice second
run:
- cmd: echo 'The $CHOICE choice is __MATCH__'
apply:
- fn: match
args:
var: *choice
token: __MATCH__
choices: ["first", "second", "third"]
values: ["bird", "cat", "dog"]
Result:
parent_dir
Replaces a token by the path to the parent directory of a given path.
| Name | Type | Description |
|---|---|---|
token |
str |
Token to be replaced. |
path |
str |
Path whose parent directory will be extracted. |
n |
int |
Repeat this function recursively n times. |
Example:
recipe:
step:
env:
DIR: &dir /path/to/parent_dir/file.txt
run:
- cmd: echo 'The parent dir of $DIR is __PARENT_DIR__'
apply:
- fn: parent_dir
args:
token: __PARENT_DIR__
path: *dir
Result:
randchoice
Picks a random choice from a list.
| Name | Type | Description |
|---|---|---|
token |
str |
Token to replace. |
choices |
List[str] |
List of str to choosen from. |
seed |
int | None |
Random seed. |
Example:
recipe:
step:
run:
- cmd: echo 'My pet is a __CHOICE__'
apply:
- fn: randchoice
args:
token: __CHOICE__
choices: ["bird", "cat", "dog"]
Result:
randint
Generates a random integer between min (included) and max (included).
| Name | Type | Description |
|---|---|---|
token |
str |
Token to replace. |
min |
int |
Minimum integer number to generate. |
max |
int |
Maximum integer number to generate. |
seed |
int | None |
Random seed. |
Example:
recipe:
step:
run:
- cmd: echo 'The lottery winner is the number __WINNER__'
apply:
- fn: randint
args:
token: __WINNER__
min: 0
max: 100
Result:
randfloat
Generated a random float-point number within the [min, max) range.
| Name | Type | Description |
|---|---|---|
token |
str |
Token to replace. |
min |
int |
Minimum floating-point number to generate (inclusive). |
max |
int |
Maximum floating-point number to generate (exclusive). |
seed |
int | None |
Random seed. |
Example:
recipe:
step:
run:
- cmd: echo 'The generated floating-point number is __FLOAT__'
apply:
- fn: randfloat
args:
token: __FLOAT__
min: 0.0
max: 1.0
Result:
replace
Replace one or multiple tokens by a given value.
| Name | Type | Description |
|---|---|---|
**kwargs |
str |
Keyword arguments where each key is a token and each value is the token value. If a list of multiple values is passed, one command per value will be generated. |
Example:
recipe:
step:
run:
- cmd: echo 'The dog arrived first, and then the cat came'
apply:
- fn: replace
args:
dog: bird
cat: fish
Result:
Example (multiple values):
recipe:
step:
run:
- cmd: echo 'The dog arrived first, and then the cat came'
apply:
- fn: replace
args:
dog: [bird, lizard]
cat: [fish, wizard]
Result:
repeat
Repeats a command a given number of times.
| Name | Type | Description |
|---|---|---|
n |
int |
Number of times to repeat the command. |
Example:
Result:
run_if
Runs a command only if a variable matches a given value.
| Name | Type | Description |
|---|---|---|
var |
str |
Number of times to repeat the command. |
value |
str |
Expected value to run the command. |
Example:
recipe:
step:
env:
RUN: !prompt
message: Run command?
choices: ["y", "n"]
default: "y"
run:
- cmd: echo 'The command has been executed'
apply:
- fn: run_if
args:
var: $RUN
value: y
Result:
shard
Keeps only one shard of the commands, so that they can be split across multiple processes or jobs. Commands are assigned to shards in a round-robin fashion.
| Name | Type | Description |
|---|---|---|
index |
int |
Index of the shard to keep, from 0 to num - 1. |
num |
int |
Total number of shards. |
Example:
recipe:
step:
run:
- cmd: echo 'Processing file __FILE__'
apply:
- fn: dir_files
args:
token: __FILE__
dir: corpus
- fn: shard
args:
index: $SLURM_ARRAY_TASK_ID
num: 20
When run as a SLURM array job with sbatch --array=0-19, each of the 20 jobs processes a different 1/20 of the files.
Skipping existing outputs
A dict command can contain a skip_if_exists key with one or more paths. The command is skipped if all of them exist. The same tokens replaced by the modifiers in cmd are also replaced in skip_if_exists, so each generated command can point to its own output. Relative paths are resolved from the cwd of the step.
recipe:
step:
run:
- cmd: ffmpeg -i __FILE__ out/__STEM__.flac
skip_if_exists: out/__STEM__.flac
apply:
- fn: dir_files
args:
token: __FILE__
dir: corpus
stem_token: __STEM__
If the recipe stops halfway through, running it again only processes the files that are missing. Skipped commands are shown as Skipped: in the output. Note that an output left incomplete by an interrupted command also counts as existing.
Parallel commands
By default, the commands of a step are run one after another. The parallel key runs up to that number of commands of the step at the same time, or one per CPU core if set to auto. Steps are still run one after another.
recipe:
step:
parallel: 8
run:
- cmd: ffmpeg -i __FILE__ out/__STEM__.flac
apply:
- fn: dir_files
args:
token: __FILE__
dir: corpus
stem_token: __STEM__
If a command fails, no new commands are started, and recp stops once the running commands finish. With --ignore-errors, all commands are run.
The --jobs option overrides the parallel key of all steps that have one. For example, --jobs 1 runs all commands one after another, which is useful for debugging.
Steps without a parallel key, or run with --jobs 1, are not affected by anything in this section: their commands write directly to the terminal, as usual.
Output modes
The outputs of parallel commands are never mixed. How they are shown is set with the --output option:
| Mode | Description |
|---|---|
auto |
Default. Uses board when printing to a terminal, and grouped otherwise (e.g. when the output is redirected to a file or in CI). |
board |
Shows a live table under the output with the status, run time and last output line of each command. Progress bars of the commands are shown as they update. The full output of each command is printed above the table when it finishes. |
grouped |
Prints the full output of each command when it finishes, with no live table. |
prefix |
Prints output lines as they arrive, prefixed with the ID and label of their command. Progress bars are printed at most once every few seconds. |
In board mode, a step that runs a model on four datasets, run with --jobs 2, looks like this while running:
[1/1] Running step 'infer' ...
Parallel: 2 commands at a time
Logs: ~/.local/state/recp/runs/20261006-111502-4512
──────────────────────────────────────────────────────────────────────────────
infer 1 done · 2 running · 1 queued · 0 failed 04:07
# Status Time Label Last line
1 Done 03:12 --dataset clean
2 Running 04:07 --dataset noisy Processing: 58%|█████▌ | 464/800
3 Running 00:55 --dataset reverb Processing: 13%|█▎ | 104/800
4 Queued - --dataset music
──────────────────────────────────────────────────────────────────────────────
When the step finishes, the table is replaced by a summary of all commands. Steps with more than 20 commands only list the commands that did not succeed.
With --output prefix, the same run looks like this:
[1 --dataset clean] Loading model ...
[2 --dataset noisy] Loading model ...
[1 --dataset clean] Processing: 10%|█ | 80/800
Command labels
Each parallel command gets a short label to tell it apart from the others. By default, the label is the part of the command that differs from the other commands of the step, such as --dataset noisy in the example above. Labels are only inferred when all commands of the step have the same number of arguments, so a step with unrelated commands shows the commands themselves.
A label key sets the label explicitly. Tokens replaced by the modifiers in cmd are also replaced in label:
run:
- cmd: python infer.py --input data/__SET__ --output out/__SET__
label: __SET__
apply:
- fn: cartesian_product
args:
__SET__: [clean, noisy, reverb, music]
Info
The label key, the live table and the recp status, recp log and recp follow commands were introduced in version 0.4.0, so update your minimum_required_version if a recipe uses label and that key is present.
Following parallel commands
The output of every parallel command is saved to a log file. The Logs: line printed at the start of the step shows the folder of the current run. Commands are numbered from 1 within each run, across all of its parallel steps.
These commands work from another terminal while the recipe is running, or after it finishes:
recp status # status of the commands of the latest run
recp log 4 # full command and output of command 4
recp log 4 -n 20 # last 20 lines of output of command 4
recp follow 2 # stream the output of command 2 while it runs
recp follow waits for the command to start if it is still queued, and stops when the command finishes. Pressing Ctrl+C stops following, but not the command. Its exit code is the exit code of the command.
All three take a --run option to show an older run. To list the stored runs, use:
The logs of the last 20 runs are kept. This number can be changed with:
Recipe shortcuts
You can store frequently used recipes in a folder that recp reads automatically, and then refer to a recipe just by its name.
To add a recipe to the recipes folder, use:
Then this recipe can be run as:
You can see the folder location and other recp settings by running:
To change the default recipes folder you can use:
To list the recipes in the recipes folder, you can use:
Editor support
A JSON Schema for recipe files is available. With the YAML extension for VS Code, or any editor using the YAML language server, it provides autocompletion and validation of recipe files. To enable it, add the following line at the top of your recipe:
# yaml-language-server: $schema=https://raw.githubusercontent.com/eagomez2/recp/main/recipe.schema.json
To avoid warnings about the recp constructors, add them to your VS Code settings: