.neuro/<batch-name>.yml
or .neuro/<batch-name>.yaml
file under the project's root. The config filename should be lowercase and not start with a digit if the id
attribute is not specified. The following YAML attributes are supported:kind
batch
for batch workflows.id
id
is generated from the filename of the workflow config, with hyphens being (-
) replaced with underscores (_
). It's available as a ${{ flow.flow_id }}
in experssions.title
defaults
defaults.env
tasks.env
.neuro-flow
uses the most specific environment variable. For example, an environment variable defined in a task will override the workflow default.defaults.life_span
tasks.life_span
. If not set, the default task lifespan is 1 day. The lifespan value can be one of the following:float
number representing the amount of seconds (3600
represents an hour)1d6h15m
(1 day, 6 hours, 15 minutes)365d
). Keep in mind that this may be dangerous, as a forgotten job will consume cluster resources.defaults.preset
tasks.preset
. The system-wide default preset is used if both defaults.preset
and tasks.preset
are omitted.defaults.volumes
defaults.schedule_timeout
float
number representing the amount of seconds (3600
represents an hour)1d6h15m45s
(1 day, 6 hours, 15 minutes, 45 seconds).default.schedule_timeout
and tasks.schedule_timeout
are omitted.defaults.tags
tasks.tags
.defaults.workdir
tasks.workdir
for more information.defaults.fail_fast
true
, the system cancels all in-progress tasks if any one of them fails. It can be overridden by tasks.strategy.fail_fast
. This is set to true
by default.flow
context, params
context, env
context, tags
context, volumes
context, images
context.defaults.max_parallel
flow
context, params
context, env
context, tags
context, volumes
context, images
context.defaults.cache
defaults.cache.strategy
"none"
Don't use caching at all."default"
The basic caching algorithm that reuses cached outputs in case task definitions and all expression contexts available in the task's expressions are the same."default"
flow
context, params
context, env
context, tags
context, volumes
context, images
context.defaults.cache.life_span
float
number representing the amount of seconds (3600
represents an hour)1d6h15m45s
(1 day, 6 hours, 15 minutes, 45 seconds)14d
(two weeks).neuro-flow
will ignore cache entries that were added longer ago than the new cache.life_span
value states.flow
context, params
context, env
context, tags
context, volumes
context, images
context.images
.force_rebuild
flag is not set, then Neu.ro Flow will not attempt to build the image from scratch. If this flag is set or the image is not in the registry, then the platform will start buliding the image.images
section is not required. A task can specify the image name in a plain string without referring to the ${{ images.my_image.ref }}
context.images.<image-id>
image-id
is a string and its value is a map of an image's configuration data. You must replace <image-id>
with a string that is unique to the images
object. <image-id>
must start with a letter and contain only alphanumeric characters or underscore symbols _
. Dash -
is not allowed.images.<image-id>.ref
hash_files
). Its value will be calculated before the remote executor starts.images.<image-id>.context
${{ flow.workspace / 'some/dir' }}
) or a remote volume spec (e.g. storage:subdir/${{ flow.flow_id }}
). If it's a local path, it cannot use anything that's not available at the beginning of a bake (for example, action inputs). If it's a remote path, usage of dynamic values is allowed. Local context will be automatically uploaded to storage during the "local actions" step of the bake.neuro-flow
cannot build images without the context, but can address pre-built images using images.<image-id>.ref
​images.<image-id>.dockerfile
Dockerfile
by default..context
- if it's a local path, dynamic values are forbidden and it's automatically uploaded. If it's a remote path, then dynamic values are allowed.images.<image-id>.build_args
images.<image-id>.env
images.<image-id>.volumes
images.<image-name>.force_rebuild
params
neuro-flow bake
command introspection, shell autocompletion, and generation of more detailed error messages:--param
.--meta-from-file
.images
neuro-flow build <image-id>
.images
section is not required. A task can specify the image name in a plain string without referring to the ${{ images.my_image.ref }}
context.images.<image-id>.context
images.<image-id>.dockerfile
images.<image-id>.build_args
images.<image-id>.env
images.<image-id>.volumes
images.<image-id>
image-id
is a string and its value is a map of the task's configuration data. You must replace <image-id>
with a string that is unique to the images
object. <image-id>
must start with a letter and contain only alphanumeric characters or underscore symbols _
. Dash -
is not allowed.images.<image-id>.ref
tasks.image
expression.volumes
neuro-flow upload
and neuro-flow download
commands. They can only be mounted to a task by using task.volumes
attribute.volumes.<volume-id>.local
volumes.<volume-id>
volume-id
is a string and its value is a map of the volume's configuration data. You must replace <volume-id>
with a string that is unique to the volumes
object. The <volume-id>
must start with a letter and contain only alphanumeric characters or underscore symbols _
. Dash -
is not allowed.volumes.<volume-id>.remote
volumes.<volume-id>.mount
volumes.<volume-id>.read_only
tasks
tasks.needs
. To start execution, run neuro-flow bake <batch-id>
.tasks.id
tasks.needs
. The value must start with a letter and only contain alphanumeric characters or underscore symbols (_
). The dash symbol (-
) is not allowed.task-<num>
in the command line output. Here, <num>
is an index from the tasks
list.tasks.needs
tasks.enable
statement that causes it to ignore the dependency failure.
By default, tasks.needs
is set to the previous task in the tasks
list. In case the previous task has matrix
enabled, the current task will only run after all matrix tasks are completed.
This property also specifies what entries are available in the needs context.needs
.running
and completed
.neuro-flow inspect --view BAKE_ID
to view the graph of running batch tasks converted to a PDF file.tasks.enable
${{ success() }}
flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.strategy
tasks.strategy.matrix
matrix
attribute defines a set of configurations with which to run a task. Each key in this mapping sets some variable that will be available in the matrix
context in expressions of the corresponding task. Each value should be an array, and neuro-flow
will start task variants with all possible combinations of items from these arrays. The matrix can generate 256 different tasks at most.example_a
and example_b
will be generated.a_x
, a_y
, b_x
, b_y
will be generated.-<param-1>-<param-2>
tasks.strategy.matrix.exclude
exclude
is a list of combinations to remove from the matrix. These entries can be partial, in which case all matching combinations are excluded.a_x_2
, a_y_1
, a_y_2
, b_x_1
, b_x_2
will be generated.tasks.strategy.matrix.include
include
is a list of combinations to add to the matrix. These entries cannot be partial. In case exclude
is also set, include
will be applied after it.a_x
, a_y
, b_x
, b_y
, a_z
will be generated.tasks.strategy.fail_fast
true
, the system cancels all in-progress tasks if this task or any of its matrix tasks fail. Default: true
.flow
context, params
context, env
context, tags
context, volumes
context, images
context, strategy
context (contains flow global values).tasks.strategy.max_parallel
flow
context, params
context, env
context, tags
context, volumes
context, images
context, strategy
context (contains flow global values).tasks.cache
tasks.cache.strategy
"none"
Don't use caching at all"default"
The basic caching algorithm that reuses cached outputs in case task definition and all expression contexts available in task expressions are the same."inherit"
flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context.tasks.cache.life_span
float
number representing an amount of seconds1d6h15m45s
(1 day, 6 hours, 15 minutes, 45 seconds)neuro-flow
will ignore cache entries that were added longer ago than the new cache.life_span
value specifies.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context.tasks.image
python:3.9
or ubuntu:20.04
) or on the Neu.ro Registry (image:my_image:v2.3
).images
section:flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.cmd
cmd
, bash
, and python
are mutually exclusive: only one of the three is allowed at the same time. If none of these three attributes are specified, the CMD
from the tasks.image
is used.cmd
attribute points to the command with optional arguments that is available in the executed tasks.image
.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.bash
bash
script to run.cmd
to run a bash script can be tedious: you need to apply quotas to the executed script and set proper bash flags to fail on error.bash
attribute is essentially a shortcut for cmd: bash -euo pipefail -c <shell_quoted_attr>
.cmd
, bash
, and python
are mutually exclusive.bash
should be pre-installed on the image to make this attribute work.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.python
python
script to run.python
instead of bash
, this notation is great for you.python
attribute is essentially a shortcut for cmd: python3 -uc <shell_quoted_attr>
.cmd
, bash
, and python
are mutually exclusive.python3
should be pre-installed on the image to make this attribute work.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.entrypoint
ENTRYPOINT
if needed or specify one. Unlike the Docker ENTRYPOINT
instruction that has a shell and exec form, the entrypoint
attribute only accepts a single string defining the executable to be run.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.env
neuro-flow
uses the most specific environment variable. For example, an environment variable defined in a task will override the workflow default.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.http_auth
flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.http_port
80
port as an HTTPS-protected resource.0
to disable the feature entirely.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.life_span
3600
for an hour)1d6h15m
(1 day, 6 hours, 15 minutes)365d
) for lifespan-disabling emulation. Keep in mind that this can be dangerous, as a forgotten task will consume cluster resources.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.name
neuro
tool.neuro-flow
tool doesn't require it to work properly.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.pass_config
true
if you want to pass the Neu.ro config used to execute neuro-flow run ...
command into the spawned task. This can be useful if you use a task image with Neuro CLI installed and want to execute neuro ...
commands inside the running task.flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.preset
flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.schedule_timeout
float
number representing an amount of seconds1d6h15m45s
(1 day, 6 hours, 15 minutes, 45 seconds)flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.tags
project:<project-id>
, flow:<flow-id>
, and task:<task-id>
).flow
context, params
context, env
context, tags
context, volumes
context, images
context, matrix
context, strategy
context, needs
context.tasks.title
<task-id>
if not overridden.