anoopkc

zenv: project environments on laptops and HPC clusters

Wed, 14 Oct 2026 18:14:11 +0000

zenv: project environments on laptops and HPC clusters

Mean time per command on an Apple Silicon Mac and an HPC system, as hyperfine means with zenv v0.28.1. The HPC system is a shared JURECA login node with 256 CPUs, and the project is on GPFS.

Before a Python project runs on JURECA, one of the clusters at the Jülich Supercomputing Centre, the shell needs three things: the site’s modules for the compiler stack including Python, a virtual environment built on that Python, and the variables the project reads. By hand, that is

module load Stages/2026 StdEnv Python
source ~/sim/.venv/bin/activate
export SIM_DATA=/p/scratch/myproject/sim/data

typed again in every new shell and copied into every batch script(+ a one time cost of setting up the environment). The order matters as well: the venv’s Python is the module’s Python, and without the modules loaded it does not start at all, because it cannot find its own libpython3.13.so. And the steps are slow. On a JURECA login node, loading the modules and starting Python takes about a second, of which Python itself needs 20 ms.

zenv is a tool I wrote to keep these steps in a file next to the code, so that a project’s environment can be rebuilt on any machine and entered by name. It covers the cluster side, the modules and their slow loading, as well as the Python side, and every part is optional: on a laptop, the same tool can manage nothing but a venv.

It treates the environment as data. zenv records what activating an environment does, and replays that record in subsequent activations. The result is fast, reproducible, and usable from any shell, directory, batch script and Jupyter kernel.

It also comes with a shell hook that activates the environment when you cd into the project and deactivates it when you leave the directory. No variable pollution, no stray entries in PATH, and no typing the same three lines over and over.

One file next to the code

The three lines become an entry in zenv.json at the root of the project. A file can hold several environments, here one for a laptop and one for JURECA:

{
  "base_dir": ".zenv",
  "sim": {
    "target_machines": ["*"],
    "dependency_file": "requirements.txt",
    "env": { "SIM_DATA": "${ZENV_PROJECT_DIR}/data" }
  },
  "sim-jureca": {
    "target_machines": ["jurecadc"],
    "modules": ["Stages/2026", "StdEnv", "Python"],
    "dependency_file": "requirements.txt",
    "env": { "SIM_DATA": "/p/scratch/myproject/sim/data" }
  }
}

base_dir is where the environments are built, relative to the zenv.json file. target_machines names the machines an environment is meant for, zenv matches it against $SYSTEMNAME where a site sets it. So One can describe multiple environments that target different machines in the same file. zenv also expands variables like $HOME, ${ZENV_PROJECT_DIR} and ${ZENV_ENV_DIR} and takes everything else literally.

Build the environment once on each machine, and use it from then on:

zenv setup sim-jureca --uv            # modules, venv, requirements.txt
# --uv flag makes setup install packages with uv instead of pip. 

source $(zenv activate sim-jureca)    # activate; deactivate undoes it
zenv run sim-jureca python train.py   # one command, no activation

Setup also registers the environment in ~/.zenv/registry.json, which lets the other commands find it by name from any directory. Inside the project, . stands for the project’s environment on the current machine, so zenv run . python train.py works unchanged on the laptop and on JURECA.

For a new project, zenv init writes a zenv.json from what it finds, such as a requirements.txt or a pyproject.toml, and zenv validate reports mistakes in the file with line numbers.

Activation as data

An activation script is a small program. Each time it runs, it calls module load, which has Lmod work out the modules again, and then edits PATH and the other variables. zenv runs those steps once, during setup, and keeps the result.

To do that, setup starts one bash process that purges the modules, writes out its environment with env -0, loads the modules, and writes the environment out again. The difference between the two snapshots is the module cache: the variables the modules set, and the entries they put in front of or behind the old value of PATH, LD_LIBRARY_PATH and the other path variables. zenv stores it as a list of operations (set, prepend, append, unset), followed by the operations for the rest of activation: the venv’s VIRTUAL_ENV and bin/ directory, the variables from env, and a few for zenv’s own use. Together they make up zenv-env.json, the record of what activating the environment does.

Every way of entering the environment reads that record. The activation scripts for bash and zsh, csh and tcsh, and fish are generated from it, and zenv run applies it to its own process without starting a shell.

zenv setup gathers what activation does into one record, which the activation scripts and zenv run then read.

Replaying the record costs little. On a JURECA login node, zenv run starts Python from a module-based environment in 21 ms, where Python started directly takes about 20 ms. With "module_cache": false, zenv calls Lmod on every use instead, and the same command takes about a second. Activation in an interactive shell gains in the same way, since with the cache the generated script does not call module: in bash it takes 9 ms with the cache and about a second without.

Because zenv knows exactly what activation changed, it can undo it. deactivate gives each variable its previous value back, or unsets it if it had none, and activating a second environment deactivates the first. Switching back and forth leaves no stray entries in PATH.

The record also makes the environment usable where nobody types source. zenv run replaces itself with the command it starts, so the exit status and signals are the command’s own, which is what a batch script expects from zenv run sim-jureca python train.py. Jupyter kernels made with zenv jupyter create sim-jureca start through zenv run as well. A notebook then gets the environment’s modules even when the Jupyter server was started with other ones, and the Interrupt button reaches the Python process. The environment needs ipykernel for this.

Two things cannot be recorded. Hooks in activate.commands or activate.script are shell code, so an environment that has them is run by sourcing its activate.sh in bash; plain variables belong in env, where they stay data. And the module cache does not notice when the site changes its modules. It is tied to the cluster through $SYSTEMNAME, so it is reused on every node of JURECA and never replayed on another system, but after a module is updated during maintenance, run zenv setup again to capture it afresh.

Python on top of the modules

On a laptop the Python part is most of what zenv does. On a cluster it has to fit around the modules.

zenv picks the interpreter in a fixed order: a Python from the loaded modules first, then the fallback_python named in zenv.json, then a Python installed and pinned with zenv python install and zenv python pin, then the system’s python3. If the pyproject.toml that setup reads declares requires-python, setup checks the chosen interpreter against it before creating anything, and when it does not match, says which of these to change.

zenv writes the venv itself. A venv is a pyvenv.cfg file, bin/python links to the base interpreter and empty site-packages directories, and zenv creates them directly, without python -m venv or ensurepip. It then installs pip into the venv, with uv when you pass --uv. A new environment without packages is ready in 115 ms on a Mac, where python -m venv alone takes 1.5 s.

Modules often ship Python packages of their own, such as numpy or mpi4py built against the site’s libraries. Before installing, zenv reads the package metadata in the directories on the modules’ PYTHONPATH, the same *.dist-info and *.egg-info entries that importlib.metadata reads, and leaves out every requirement a module already provides, listing what it left out. --force installs them anyway.

New dependencies go in with zenv add, which installs a package and then records it in the project’s manifest:

zenv add . "xarray>=2024.1" --uv   # recorded in the manifest
zenv add . pytest --dev --uv       # as a development dependency

Where the package goes depends on what the environment uses: a line appended to requirements.txt, an entry in [project].dependencies (or [dependency-groups].dev with --dev) of pyproject.toml, where zenv rewrites only that entry and leaves comments alone, or the dependencies list in zenv.json. The manifest changes only after the install succeeds, so a misspelled package name leaves the files as they were.

From any directory

Registered environments can be used by name wherever you are. zenv ls lists the ones for the current machine, cd $(zenv cd sim-jureca) goes to the project, and zenv alias create gives an environment a second name.

Aliases also help on shared filesystems. JURECA and JUWELS mount the same project directories, so one checkout serves both, with an environment for each machine in its zenv.json. Give both environments the same alias, and the alias picks the one whose target_machines match the machine you are on:

zenv alias create hpc sim-jureca
zenv alias create hpc sim-juwels
zenv run hpc python train.py    # sim-jureca on JURECA, sim-juwels on JUWELS

. resolves the same way.

For zsh and bash there is a hook that activates a project’s environment when you cd into the project and deactivates it when you leave, much as direnv does:

eval "$(zenv hook zsh)"    # in ~/.zshrc; for ~/.bashrc, zenv hook bash

It activates only environments you have set up yourself, so the zenv.json in a freshly cloned repository does nothing until you run zenv setup.

zenv also keeps an environment in step with its configuration. zenv run, zenv activate and zenv cd compare the environment’s entry in zenv.json, and the dependency and module files it names, with their state at the last setup. If anything changed, they rebuild first, with the flags of that setup, so --uv sticks. Add a line to requirements.txt, and the next zenv run installs the package before it runs your command. When the module list has not changed, the rebuild replays the module cache instead of calling Lmod again; on JURECA that brought the setup of a three-module environment from 1.25 s down to 0.16 s. The cd hook is the exception. It never rebuilds, so that cd stays fast, and prints a line saying the environment is out of date instead.

Environments without Python

Nothing above requires Python. A C++ project can use zenv for its compilers and variables alone:

{
  "base_dir": ".zenv",
  "tools": {
    "target_machines": ["jurecadc"],
    "python": false,
    "modules": ["Stages/2026", "GCC", "CMake"],
    "env": {
      "PATH": { "prepend": "${ZENV_PROJECT_DIR}/scripts" },
      "OMP_NUM_THREADS": "8"
    }
  }
}

Setup loads and caches the modules, creates an empty bin/ directory that activation puts on PATH, and runs any setup hooks. There is no venv, and activation does not set VIRTUAL_ENV, so uv and pip will not mistake the environment for one. Setting "python": true later adds a venv in the same directory and keeps the modules, the hooks and the files in bin/.

How fast it is

zenv is a single binary written in Zig, statically linked on Linux, and needs no Python to run, so it works on a machine before any Python is set up. Its own commands cost about as much as starting an empty process. The time you notice comes from the programs it starts: uv, Lmod and Python.

Mean time (ms)	Mac	HPC system
`true`, an empty process	1.2	2.1
`zenv ls`	1.9	2.0
`zenv activate <env>`	2.0	2.2
`zenv run <env> true`	3.1	3.9
`bash -l`, a login shell alone	–	495
`source $(zenv activate <env>)` in a login shell, with modules	–	525
`module load` and `source .venv/bin/activate` by hand, in a login shell	–	1,586
`zenv setup <env> --uv`, nothing changed	21	166
`zenv setup <env> --uv`, new environment without packages	115	358
`python -m venv`, for comparison	1,456	2,618

The numbers are hyperfine means for zenv 0.28 on an Apple Silicon Mac and on an HPC system, a JURECA login node (256 CPUs, project on GPFS). The login node is shared with other users, so its numbers vary by about 0.8 ms. The three login-shell rows need modules, so the Mac has none; they come from an earlier run with zenv 0.27.8, whose zenv activate took 1.9 ms longer.

Getting it

zenv runs on Linux and macOS, on x86_64 and aarch64:

curl -fsSL https://raw.githubusercontent.com/anoopkcn/zenv/HEAD/install.sh | bash
# or
brew install anoopkcn/zenv/zenv

The source and the full reference, with every key of zenv.json and the rules the cd hook and the module cache follow, are at github.com/anoopkcn/zenv.

Dimensionality reduction: from pixels to a picture, a summary or a short list

Sun, 27 Sep 2026 12:57:21 +0000

Dimensionality reduction: from pixels to a picture, a summary or a short list

This is the written version of a 45-minute talk I gave in September 2026 on data preparation and dimensionality reduction. Every data figure except the curse-of-dimensionality plot is computed from the tabletop dataset, and the numbers in the text come from the same runs.

One object on a table

Every example in this post uses the same data: tabletop [2], a set of synthetic photos of a single object standing on a table. A small ray caster renders each photo, so every photo comes with the true values of the seven factors that produced it:

Factor	Levels
Shape	4 (cube, sphere, cylinder, capsule)
Colour	12
Size	8
Rotation	12
Position	8
Light	4
Table	2 (beige, grey-blue)

Two cameras look at each scene. One takes a colour photo, the other a depth image that stores the distance from the camera along each pixel’s ray. Both are 64 x 64 pixels.

Six scenes from tabletop. Top row: the colour photo. Bottom row: the depth image of the same scene.

The full grid of factors gives 294,912 photos. I use a subset of 6,144 that is still a complete grid on its own: 4 colours (red, lime, cyan and violet), 4 rotations (0°, 90°, 180° and 270°), 3 positions, 2 lights, and every shape, size and table.

The depth camera sees shape, size, rotation and position. It cannot see colour, light or the table, because none of those change a distance. That detail comes back when we join the two cameras.

What dimensionality reduction does

A colour photo of 64 x 64 pixels with three channels is 12,288 numbers. Apart from a little camera jitter and pixel noise, only seven things change from one photo to the next. If you had to describe a photo in as few numbers as possible and knew the factors, seven would do.

Each photo is one row of p = 12,288 numbers. Dimensionality reduction describes it with l numbers instead.

Formally, you start with $n$ samples, each described by $p$ features, and end with $l$ features per sample, where $l \ll p$ . Some information is always lost on the way. Each method makes its own choice of what to keep, and most of this post is about those choices.

The reduced data are useful in three ways. You can look at them: scatter plots, heatmaps and quality control need two or three dimensions. You can model with them: clustering, classification and trajectories work better on a few informative features. And they make computing cheaper: less storage, faster training and fewer parameters to fit.

A plot of two principal components is easier to read than 12,288 numbers, but each component mixes thousands of pixels, so the plot does not tell you which pixels matter. Feature selection is the exception, because it keeps original measurements you can point at.

The curse of dimensionality

High-dimensional space behaves in ways that 2-D intuition does not prepare you for. The plot below places 500 uniformly random points in a cube of growing dimension. For 20 query points, it compares the distance to the nearest point with the distance to the farthest.

The ratio of nearest to farthest distance for uniform random points. At 12,288 dimensions, the size of one photo, it reaches 0.97.

In two dimensions the nearest point is close compared with the farthest. At 12,288 dimensions the nearest point is 97% as far away as the farthest one [3]. When every point is about equally far from every other, neighbourhoods and clusters lose their contrast.

Real data are not uniform, and that is what makes reduction possible. The tabletop photos occupy a thin, roughly seven-dimensional sliver of the 12,288-dimensional pixel space. Every method below is a different way of finding that sliver.

Three ways to get from p features to l

The methods fall into three families. What separates them is what happens to the original features.

Transformation mixes all features into new ones. Aggregation summarizes groups. Selection keeps a subset.

Transformation builds new features, each a mix of all the originals: PCA, ICA, t-SNE, UMAP and autoencoders.
Aggregation splits the features into groups and summarizes each group with one number: pixel blocks, gene sets, clusters found by $k$ -medoids.
Selection keeps a subset of the original features: ReliefF, forward selection, the elastic net.

Only selection keeps measurements you can still point at, in our case single pixels. The other two trade that for compactness.

There is also a rough split by supervision. Transformation and aggregation usually run without a target variable. Selection usually keeps whatever predicts a target. The split is only a tendency: linear discriminant analysis is a supervised transformation, and dropping low-variance features is unsupervised selection.

Preparing the data

Before any method runs, the photos have to become a matrix of numbers on a common scale, and the data have to be split.

One step of a longer pipeline

Dimensionality reduction sits in the middle of a longer chain of steps, and the order of those steps matters.

A typical pipeline. The split into training and test data comes before every step that learns from the data.

Missing-value imputation, scaling and dimensionality reduction all learn something from the data: a mean, a standard deviation, a set of components. If they run before the split, the test data have already shaped them. So the split comes before them, at step 5, and steps 6 to 8 are fitted on the training data alone.

From image files to an analysis matrix

The tabletop photos come in an HDF5 file. Turning them into a matrix with one row per photo takes a few lines:

with h5py.File("smoke.h5") as f:
    images = f["images"][:]      # uint8,  (6144, 64, 64, 3)
    depth  = f["depth"][:]       # uint16, (6144, 64, 64)
    labels = f["labels"][:]      # int8,   (6144, 7): one level per factor

X = images.astype(np.float32) / 255        # integers 0-255 -> numbers 0-1
X = X.reshape(len(X), -1)                  # (6144, 12288): one row per photo
D = depth.astype(np.float32) * 8 / 65535   # stored steps -> scene units
D = D.reshape(len(D), -1)                  # (6144, 4096)

Three things happen here:

The types change. Pixels, depths and labels all arrive as small integer types. Anything used in arithmetic becomes a float first, for a reason the next section shows.
The photos are flattened. Each colour photo becomes one row of 12,288 columns, and each depth image one row of 4,096. This throws away which pixel sits next to which. Methods that use the neighbourhood, such as convolutional networks, keep the 2-D shape instead.
The labels stay out of X. The seven factors are what we hope to recover. They colour the plots and serve as targets, but they are never inputs.

Integer types overflow without warning

Both of the bugs below came up while I was preparing the figures for this talk.

Expression	Expected	Got
`(depth * 8 / 65535).max()`	8.0	1.0
`labels[:, 1] * 30` for cyan	180	−76
`labels[:, 1] * 30` for violet	270	14

depth is uint16, which holds 0 to 65,535. The file spreads distances from 0 to 8 scene units over those steps, so multiplying by 8 and dividing by 65,535 turns a step count back into a distance. Multiplying the largest value by 8 does not fit in 16 bits, so NumPy wraps the product around and the depth range comes out as 0 to 1 instead of 0 to 8. labels is int8, which holds −128 to 127. The colour label, labels[:, 1], is a hue level from 0 to 11, and the twelve hues sit 30 degrees apart on the colour wheel, so the level times 30 gives the hue angle: 180 for cyan and 270 for violet. Both wrap, to 180 − 256 = −76 and 270 − 256 = 14, and the plot colours come out wrong. Neither case raises an error.

The fix is to convert first, with .astype(np.float32). The habit that catches these bugs is to print each array’s dtype, minimum and maximum right after loading.

Scale before you reduce

Suppose you join the two cameras into one matrix, so each row holds the colour photo followed by the depth image:

Camera	Features	Range	Share of total variance
Colour	12,288	0 to 1	13%
Depth	4,096	1.8 to 8	87%

Most reduction methods follow the largest variance. Joined as stored, depth holds 87% of the variance with a third as many features, so depth decides what the method sees.

1,500 random photos in the first two principal components, coloured by the table they stand on. Left: the cameras joined as stored. Right: each camera scaled first.

In the colour photos alone, the table is the largest source of variation. The depth camera cannot see the table at all. Joined unscaled, the table vanishes from the top components (left panel). With both cameras scaled to the same total variance, it comes back as the second component (right panel).

There are two common fixes:

Standardize each feature, subtracting its mean and dividing by its standard deviation: $z_{ij} = (x_{ij} - \bar{x}_j) / s_j$ .
Scale each block (camera, assay, instrument) to the same total variance.

Standardizing works column by column, so it gives the same result before or after the join. Block scaling needs to know which columns came from which camera.

Standardizing is the usual fix when features carry different units: centimetres and kilograms, gene counts and protein levels. For pixels from a single camera it is usually wrong. It would give the dull wall pixels, which barely change, as much weight as the pixels where the object appears. Pixels from one camera already share a scale, so subtracting each pixel’s mean is enough. (Tree-based models need no scaling at all, since a split on one feature does not care about its units.)

Fit preprocessing on the training data only

Imputation, scaling, dimensionality reduction and feature selection all learn from data. Fit them on the training samples, and only apply them to the test samples:

pca = PCA(n_components=50)
Z_train = pca.fit_transform(X[train])   # learns the mean and the components
Z_test  = pca.transform(X[test])        # reuses them, learns nothing

The classic mistake is to choose the top features on all samples and then cross-validate a classifier on those features. The accuracy looks excellent and does not reproduce, because the test samples helped choose the features. scikit-learn’s Pipeline [4] makes the correct version the easy one to write.

Near-identical samples leak as well. A cube one size step larger looks almost the same as the smaller cube, so a random split puts near-copies of each test photo into the training set. Holding out whole groups avoids that. The feature-selection section below holds out one object colour at a time for this reason.

For plots with no test set, fitting on all samples is fine.

Feature transformation

Principal component analysis

Start with two pixels, three rows apart, across all 6,144 photos. They move together, with a correlation of 0.92.

Two pixels three rows apart. The first principal component points along the diagonal and carries 96% of their joint variance.

Principal component analysis (PCA) [5] finds the direction of largest variance in the data, then the direction of largest remaining variance at right angles to the first, and so on. For centred data $X$ , the first component is the unit vector

$w_1 = \arg\max_{\lVert w \rVert = 1} \operatorname{Var}(Xw)$

For these two pixels, the first component alone carries 96% of the variance. With 12,288 pixels the same redundancy repeats in many directions at once.

With two pixels you can draw the original axes and the components on top of them, as above. With 12,288 pixels you cannot, so you plot the photos along two components instead. Each point is one photo, placed by its score on each component: the weighted sum of its pixels, $z_1 = x w_1$ . The scaling figure earlier is such a plot. PC1 against PC2 shows the most variation of any pair, but for these photos that is only 34% (22% and 12%, as the next figure shows), so other pairs are worth plotting too. Every axis mixes all the pixels, so it means nothing on its own. You find out what it tracks by colouring the points by a known factor, as the scaling figure does with the table, or by drawing the component as an image.

Each component is a weighted sum of all the features. PCA needs no labels, runs fast, gives the same answer every time, and can project new samples onto the components it has learned. It is the usual first look at any new dataset.

The largest variance is not always the variation you care about. It may be technical, or a factor you are not studying, as the next figure shows.

For images, each component is an image

A component has one weight per feature. Here every feature is a pixel, so the 12,288 weights can be put back into the 64 x 64 x 3 grid the pixels came from and shown as a picture. This works only because pixels have a fixed place in a grid. For most data it does not: if the features are age, weight and blood pressure, the weights are just a list. The picture is easier to read added to the mean photo: start from the mean and add or subtract a multiple of the component, and you see what changes along it.

The mean photo minus (top) and plus (bottom) three standard deviations of each of the first six components. Above each column: its share of the variation, and the factor that sorts the photos along it.

The number above each column is the component’s share of all the variation in the photos. The word below it says what the component tracks. To find it, I gave every photo a score along the component and checked which of the seven factors sorts those scores best. Along PC1, the beige-table photos sit at one end and the grey-blue ones at the other, with almost no overlap, so PC1 is the table. PC2 and PC3 move the object, PC4 changes the light, and PC6 changes the object’s colour. No single factor sorts PC5; it changes colour and position together, so not every component has a clean meaning.

The largest source of variation in these photos is the table, which is rarely what anyone wants to study.

How many components should you keep?

The cumulative explained variance tells you how much each additional component adds.

Left: cumulative explained variance. Right: photos rebuilt from their first k components.

Five components reach 50% of the variance, 17 reach 75%, and 88 reach 90%. So 90% of the variance of a 12,288-number photo fits in 88 numbers.

A variance target such as 75% or 90% is a convention. If the components feed a model, choose their number by cross-validation instead. The rebuilt photos show what the discarded components held: sharp edges and small details go first. With 20 components the position and colour of the object are already there.

Independent component analysis

Independent component analysis (ICA) [6] solves a different problem. Imagine two people talking at a party and two microphones, each recording a different mix of both voices. ICA recovers the two voices from the two recordings.

The cocktail party with photos. Two source photos are mixed twice with different weights. ICA recovers both, in its own order.

Here the voices are two photos. I mixed their table regions with weights 0.6/0.4 and 0.35/0.65, and ICA recovered both photos almost exactly. It returned them in swapped order, and another random seed could return the other order. Sign and scale are arbitrary too.

ICA looks for components that are statistically independent and non-Gaussian [7]. Independence is a stronger condition than the lack of correlation that PCA gives you. Independent components often line up with distinct underlying processes, even when those carry little variance. In the photos, one factor such as the light changes many pixels at once, and ICA tries to find one component per such source.

	PCA	ICA
Keeps	most variance	independent sources
Order	by variance	none
Good for	dominant effects	mixed signals

t-SNE and UMAP: neighbourhoods first

PCA and ICA are linear. t-SNE [8] and UMAP [9] are nonlinear methods built to preserve neighbourhoods: photos that are close in the original 12,288 numbers should stay close in the 2-D map.

The same photos in three 2-D maps, drawn in the object’s colour with a marker for its shape. t-SNE uses perplexity 30, UMAP 15 neighbours, both with fixed seeds.

To compare the three maps, I counted neighbours kept: of each photo’s 10 nearest neighbours in the original space, the share that are still among its 10 nearest neighbours in the 2-D map. PCA keeps 29%, t-SNE 63% and UMAP 60%. The measure works for any embedding, and it is worth computing before you trust the local structure of a map.

PCA spreads the photos by size and colour. t-SNE and UMAP group them by colour and shape; in the UMAP map, the size of the object changes along each arm. Perplexity, t-SNE’s main setting, is roughly the number of neighbours each point attends to.

The two methods differ in ways that matter for how you use them:

	t-SNE	UMAP
Loss	KL divergence; repulsion only through normalization	cross-entropy with explicit attraction and repulsion
New samples	no mapping	can be embedded
Output	2-D or 3-D for plots	any dimension; can feed a downstream model
Global layout	distances between clusters mean little	keeps more of it
Speed	slower	faster; approximate neighbours, worth checking on small data

Both methods find a local optimum, so a different seed can give a different layout. Run several seeds before you interpret one. A nonlinear method can follow curved structure and reveal groups that PCA smears together. You pay for that in interpretability, in stability across seeds, and for t-SNE in the ability to embed new points.

Autoencoders

An autoencoder is a neural network trained to copy its input through a narrow middle layer. The encoder compresses a photo $x$ to a small latent vector $z$ , and the decoder rebuilds a photo $\hat{x}$ from it. Training minimizes the reconstruction error $\lVert x - \hat{x} \rVert^2$ .

A convolutional autoencoder with 8 latent numbers, trained for 20 epochs on all 6,144 photos. The photo on the right is rebuilt from those 8 numbers.

The width of the latent layer sets the reduced dimension: 8 here, 2 if you want to plot it. Once trained, the encoder is a mapping, so new samples can be embedded without retraining.

With the same budget of 8 numbers per photo, the autoencoder rebuilds far better than PCA. Its reconstruction error is 22.7 per photo (squared error summed over the pixels), against 63.0 for PCA with 8 components.

The same 8 numbers per photo. PCA blurs the object away; the autoencoder keeps it.

Architecture matters at this size. A dense first layer of 128 units on a 64 × 64 × 3 photo already has 12,288 × 128 + 128 = 1,572,992 weights and biases. The convolutional autoencoder above has 380,779 parameters in total.

PCA is a special case: an autoencoder with linear layers and squared loss learns a latent space that spans the top principal components.

Variational autoencoders

A variational autoencoder (VAE) [10] changes one thing. The encoder outputs a distribution, a mean $\mu$ and a standard deviation $\sigma$ for each latent number, and $z$ is sampled from it. The loss adds a penalty that pulls each of those distributions towards a standard normal:

$\mathcal{L} = \underbrace{\lVert x - \hat{x} \rVert^2}_{\text{reconstruction}} + w \, \underbrace{\mathrm{KL}\big(\mathcal{N}(\mu, \sigma^2) \,\Vert\, \mathcal{N}(0, 1)\big)}_{\text{regularization}}$

The KL term stops the variances from shrinking to zero and the means from drifting apart. Without it, the VAE degenerates into a plain autoencoder. With it, nearby points in the latent space must decode to similar photos, so you can walk through the latent space and watch the photo change gradually.

Top: the VAE. Middle: changing only z6 changes the colour. Bottom: moving in a straight line from a red cube’s code to a red capsule’s code changes the shape.

The weight $w$ sets the balance. Too small, and the latent space has little structure; too large, and the reconstructions blur. This VAE uses $w = 1$ and 8 latent numbers. Its reconstruction error is 27.2 per photo, a little above the plain autoencoder’s 22.7. That gap is the price of the regularization.

In the middle strip, only $z_6$ changes, from −2.5 to 2.5. Across all photos, colour explains 64% of the variance of $z_6$ . Other latent numbers follow the light (63%) and the position (55%). No latent number follows shape; the best reaches 10%. So the bottom strip cannot vary one number. It moves in a straight line from the red cube’s code to the code of a red capsule that matches it in every other factor, and the shape still changes smoothly.

Ideally a VAE on tabletop would give each of the seven factors its own latent number. This one did for colour, light and position, but not for shape, the factor that changes the fewest pixels. Whether a model can separate the factors this way is called disentanglement, and it is the question tabletop was built to test.

Choosing a transformation method

Whether a method can embed new samples decides whether it can sit inside a prediction pipeline.

Method	Type	Embeds new samples	Typical use
PCA	linear	yes	first exploration, quality control, preprocessing
ICA	linear	yes	separating independent underlying signals
t-SNE	nonlinear	no	local structure, visualization only
UMAP	nonlinear	yes	visualization, and preprocessing for models
Autoencoder	nonlinear	yes	learned representations, large data

Feature aggregation

Aggregation replaces each group of features with one summary, usually the mean or median. Given groups $G_1, \dots, G_l$ of the $p$ features, the new feature $k$ of sample $i$ is

$a_{ik} = \operatorname*{mean}_{j \in G_k} x_{ij}$

Groups may overlap, and a feature may belong to no group. Turning a colour photo into a grey one is already aggregation: three channel values become one.

The groups can come from two places. Knowledge-based groups are ones you already know: blocks of neighbouring pixels, or gene sets. They are easy to interpret. Data-driven groups come from clustering the features by how they change together. They can find structure you did not expect, but a cluster may not have a name.

Correlations between pixels, sorted into the 8 clusters found by k-medoids. Pixels in one cluster correlate 0.59 on average, against 0.17 overall.

For the data-driven groups I clustered the 4,096 pixel positions with $k$ -medoids [11]. Each cluster is summarized around its medoid, a real pixel whose brightness profile best represents the group. $k$ -means would give an average profile instead, which is not any pixel you can point at.

The classic algorithm for $k$ -medoids is PAM, partitioning around medoids. It picks $k$ data points as the first medoids and assigns every point to the nearest one. The cost is the sum of the distances from each point to its medoid. PAM then tries swapping a medoid for another point, keeps the swap if the cost drops, undoes it if the cost rises, and stops when no swap helps. The animation runs PAM on 69 toy points in three blobs with $k = 3$ , few enough to follow every step.

k-medoids (PAM) on 69 toy points with k = 3. Each swap and each cost comes from the run itself. 78 seconds, no sound.

The run starts at a cost of 117.2. The first swap it tries raises the cost to 218.0 and is undone. Three of the six swaps shown are kept, and they bring the cost down to 97.1, 83.7 and finally 70.8, where no swap lowers it further. In the last part of the clip six points drift away from the blue cluster. The mean, which $k$ -means would use as the centre, follows them. The medoid stays on the same point. A medoid is always one of the data points, so PAM needs only the distances between points, and any distance will do.

PAM tries every medoid–point swap on each pass, which is slow for 4,096 pixels. CLARA [11] runs PAM on a few random samples of the pixels, assigns every pixel to the nearest medoid from each sample, and keeps the set of medoids with the lowest total cost.

Left: averaging blocks of pixels, with accuracy on an unseen object colour. Right: the 8 pixel groups found by k-medoids; each colour becomes one new feature.

Downsampling a photo is knowledge-based aggregation: we know that neighbouring pixels belong together. To measure what it costs, I trained a logistic regression to tell cubes from spheres on three object colours and tested it on the fourth, averaged over the four colours. At full resolution it gets 99% right. Averaging 4 × 4 blocks leaves 768 features, 16 times fewer, and it still gets 98% right. With 8 × 8 blocks it drops to 90%.

The pixel clusters on the right range from 131 to 1,250 pixels. They are a map of pixel positions, not a photo, and without being told, they find the wall and the regions of the table where objects stand.

Feature selection

The task: is the object a cube or a sphere?

Selection needs a target, so this section uses a classification task. I took the 1,024 photos with a cube or a sphere in the middle of the frame and turned them grey, leaving 4,096 pixels per photo. Grey is the mean of the three colour channels, so each object colour becomes a different shade of grey.

The test is whether the chosen pixels work for an object colour the model has never seen. Each fold trains on three colours (768 photos) and tests on the fourth (256 photos).

Leave-one-colour-out cross-validation. Each method is run four times, once per fold.

Holding out a colour is harder than a random split. A random split would put photos of the same object, turned by 30°, into both the training and the test set. Each method also produces four models, one per fold, and comparing them tells us how stable the selection is.

There are three families of selection methods [12]:

Filter methods score each feature first and keep the top ones.
Wrapper methods search over feature sets, scoring each set with a model.
Embedded methods select features while the model trains.

Filter methods: ReliefF

ReliefF [13] scores each pixel by how well it separates photos that are close to each other. For each photo $i$ , it finds the $k$ nearest photos in each class and updates the score of every pixel $f$ :

$\text{score}_f \mathrel{+}= \sum_{o \in \text{other}} \lvert x_{if} - x_{of} \rvert - \sum_{s \in \text{same}} \lvert x_{if} - x_{sf} \rvert$

A pixel gains score when it differs between the classes and agrees within one. Take a pixel where a cube has a corner and a sphere does not. It shows the object in cube photos and the background in sphere photos, so the first term is large, the second small, and the score high. A wall pixel looks almost the same in every photo: both terms are near zero, and so is its score.

The ReliefF score map, and the top 20 pixels drawn on a cube and a sphere.

The score map traces the outline of the cube and its shadow, which is where cubes and spheres differ. All of the top 20 pixels sit together near the lower edge of the object. They are near-copies of one pixel, and twenty copies add little over one. This redundancy is the weakness of scoring features one at a time.

ReliefF only proposes features. A separate classifier makes the prediction, here a decision tree on the top 20 pixels. It gets 98% right on the unseen lime and violet objects, 84% on red and 76% on cyan.

Wrapper methods: forward selection

Forward selection starts with no features and adds, at each step, the feature that most improves validation accuracy. To keep the search fast, the candidates here are the 256 averages of 4 × 4 pixel blocks, and the model is a decision tree. Each step tries every remaining block, trains a model on each candidate set, scores it on the three inner validation colours, and keeps the best.

The first three blocks chosen with red held out. Block 1, at the base of the object, is picked first in all four folds.

The number of features is now a hyperparameter, so it needs nested cross-validation: an inner loop chooses the features, and an outer loop tests them on the held-out colour. On the unseen colour, the selected blocks give 95% on lime and violet, 90% on cyan and 78% on red.

Greedy search is far cheaper than trying every subset, but it can miss the best combination and overfit the validation data. Backward elimination is the mirror image: start with all features and remove one at a time.

Embedded methods: the elastic net

The elastic net [14] [15] fits a logistic regression with two penalties on the pixel coefficients $w$ :

$\min_{w, c} \; C \sum_{i=1}^{n} \log\!\big(1 + e^{-y_i (x_i^\top w + c)}\big) + \rho \lVert w \rVert_1 + \frac{1 - \rho}{2} \lVert w \rVert_2^2$

The L1 penalty sets most coefficients to exactly zero, and the pixels with a non-zero coefficient are the selection. The L2 penalty keeps correlated pixels together instead of picking one of them at random. Neighbouring pixels are strongly correlated, which is the case the L2 term was designed for. This is scikit-learn’s formulation: $C$ multiplies the data term, so a smaller $C$ means a stronger penalty.

With $\rho = 0.5$ and $C = 0.05$ , the models get 96% to 100% right on the unseen colour while keeping about 210 of the 4,096 pixels each. Selection happens during training, so there is no separate search loop.

Comparing the three families

	Filter	Wrapper	Embedded
Uses a model?	no	in the loop	in the objective
Cost	low	high	moderate
Finds joint effects?	often not	yes	partly
Picks redundant features?	often	rarely	in groups
Overfitting risk	low	high	moderate
In this post	ReliefF	forward selection	elastic net

Wrappers need nested cross-validation, and embedded methods need their penalty tuned. In high dimensions a common, affordable combination is a filter to shrink the pool of features, followed by an embedded model.

Measuring selection stability

Accuracy tells you whether the selected features predict. It does not tell you whether you would select the same features again. With four folds we have four models, and we can compare what they chose.

Let model $j$ of $k$ select the positive predictors $P_j$ and the negative predictors $N_j$ . The intersections $P = P_1 \cap \dots \cap P_k$ and $N = N_1 \cap \dots \cap N_k$ hold the pixels that every model chose. The stability is

$S = \frac{1}{2} \left( \frac{1}{k} \sum_{j=1}^{k} \frac{\lvert P \rvert}{\lvert P_j \rvert} + \frac{1}{k} \sum_{j=1}^{k} \frac{\lvert N \rvert}{\lvert N_j \rvert} \right)$

$S = 1$ means every model picks the same pixels. A low $S$ means the selection depends on the split. For the elastic net on cubes and spheres, each model keeps 203 to 218 pixels, only 146 of them appear in all four, and $S = 0.69$ .

This definition has a weakness: one unusual model can empty the intersection. Nogueira et al. [16] discuss measures with better properties.

Stability separates what accuracy cannot

The next figure runs the elastic net across a range of penalties and plots accuracy and stability side by side.

Accuracy and selection stability of the elastic net as the penalty changes. Points are the mean over the four folds; bars span the minimum and maximum. The dashed line marks the best accuracy plus stability.

For cubes and spheres, compare $C = 0.05$ and $C = 0.2$ . The accuracy ranges sit on top of each other, so choosing between them on accuracy alone is choosing on noise. The stability ranges do not overlap. With four folds these are ranges rather than confidence intervals, but the difference is still clear.

Cylinder against capsule is a harder task and less stable, with stability between 0.45 and 0.56. Choosing on accuracy plus stability picks $C = 0.1$ there, at almost the same accuracy (97.7% against 97.9% for the most accurate setting).

Stability stays well below 1 because neighbouring pixels carry nearly the same information. Many different pixel sets predict equally well, and a model has no reason to prefer one of them.

Wrap-up

A starting point for your own data

Goal	Try first	Watch out for
First look, quality control	PCA on scaled data	the largest variance may not be the one you care about
Show group structure	t-SNE or UMAP	seeds, local optima
Input for a model	PCA, UMAP, autoencoder	fit inside the cross-validation
Interpretable summaries	known groups (aggregation)	only finds the groups you know
Which measurements matter	feature selection	nested cross-validation, stability

Treat the table as a place to start. Try two methods and compare them on the same data. Binary or categorical data need other methods, such as multiple correspondence analysis, logistic PCA, or nonlinear methods on a Jaccard distance.

What to take home

Prepare before you reduce. Convert the types, flatten, and scale. On tabletop, unscaled depth hides the table from PCA.
Split first. Fit scalers, reducers and selectors on the training data only.
Know what you keep. Transformation mixes features, aggregation groups them, and selection keeps a subset.
Check a nonlinear map before you trust it. Run several seeds and count the neighbours kept. t-SNE is for plots only; UMAP and autoencoders can also feed a model.
Report stability next to accuracy. Several pixel sets can predict equally well.

Resources

The course notebooks, which use a gene-expression dataset, are at github.com/HelmholtzAI-Consultants-Munich/DimRed-Course. The methods and code carry over to other data unchanged. The tabletop dataset is at github.com/anoopkcn/tabletop.

The course material is by the Helmholtz AI consultant team and licensed CC BY 4.0. tabletop is MIT-licensed.

References

Helmholtz AI consultant team. Dimensionality reduction course. CC BY 4.0. GitHub
A. Kaithalikunnel Chandran. tabletop: a complete factor grid in colour, depth and words. v2.0.0, 2026. GitHub
K. Beyer, J. Goldstein, R. Ramakrishnan, U. Shaft. When Is “Nearest Neighbor” Meaningful? ICDT 1999. doi:10.1007/3-540-49257-7_15
scikit-learn developers. Pipelines and composite estimators. scikit-learn.org
I. T. Jolliffe, J. Cadima. Principal component analysis: a review and recent developments. Phil. Trans. R. Soc. A 374, 2016. doi:10.1098/rsta.2015.0202
A. Hyvärinen, E. Oja. Independent component analysis: algorithms and applications. Neural Networks 13, 2000. doi:10.1016/S0893-6080(00)00026-5
J. Shlens. A Tutorial on Independent Component Analysis. 2014. arXiv:1404.2986
L. van der Maaten, G. Hinton. Visualizing Data using t-SNE. JMLR 9, 2008. jmlr.org
L. McInnes, J. Healy, J. Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. 2018. arXiv:1802.03426
D. P. Kingma, M. Welling. An Introduction to Variational Autoencoders. 2019. arXiv:1906.02691
L. Kaufman, P. J. Rousseeuw. Finding Groups in Data: An Introduction to Cluster Analysis. Wiley, 1990. doi:10.1002/9780470316801
I. Guyon, A. Elisseeff. An Introduction to Variable and Feature Selection. JMLR 3, 2003. jmlr.org
M. Robnik-Šikonja, I. Kononenko. Theoretical and Empirical Analysis of ReliefF and RReliefF. Machine Learning 53, 2003. doi:10.1023/A:1025667309714
H. Zou, T. Hastie. Regularization and variable selection via the elastic net. JRSS B 67, 2005. doi:10.1111/j.1467-9868.2005.00503.x
J. Friedman, T. Hastie, R. Tibshirani. Regularization Paths for Generalized Linear Models via Coordinate Descent. J. Stat. Softw. 33, 2010. doi:10.18637/jss.v033.i01
S. Nogueira, K. Sechidis, G. Brown. On the Stability of Feature Selection Algorithms. JMLR 18, 2018. jmlr.org

Agentic workflows: how coding agents work, and where you can steer them

Sat, 01 Aug 2026 12:35:28 +0000

Agentic workflows: how coding agents work, and where you can steer them

This is the written version of a talk I gave in the AI-assisted coding course in August 2026. It covers the slides of the main talk; the prompt-engineering appendix is left for another post. Course notes live at anoopkcn.github.io/ai-assisted-coding.

You type one sentence. What happens next?

Here is a typical exchange with a coding agent such as Claude Code:

> fix the failing test

• Bash(pytest -x)
  └ FAILED tests/test_auth.py::test_login
    fixture 'client' not found
• Read(tests/conftest.py)
• Edit(tests/conftest.py)  +6 lines
• Bash(pytest -x)
  └ 24 passed in 1.8s

The client fixture was never defined,
so pytest could not inject it. I added
it to tests/conftest.py; all tests pass.

You typed one sentence, and the agent ran the tests twice, read a file, edited it and explained the fix. Four questions hide in that exchange:

The model. How did it pick each word?
The context. How did it know your code?
The loop. How did it run pytest?
The harness. Who let it edit files?

Each question maps to a layer, and the layers nest. The model sits at the centre. The context is everything the model sees. The loop lets the model act on the world. The harness is the program wrapped around all of it. The rest of this post walks outward from the centre, one layer at a time.

The four layers of a coding agent. Each section of this post covers one ring.

1. The model: how is the next token chosen?

A fast pair programmer with no judgment

Think of the model as a fast, well-read colleague who has never seen your project.

What it brings	What you bring
Broad knowledge of languages and libraries	Direction: what to build and why
A first draft in seconds	Judgment: reviewing the changes
Patience for boilerplate	The final call on what ships

Under the hood the model predicts text one token at a time, by probability. It does not “understand” code the way you do.

Tokens: the units a model reads and writes

A token is a chunk of text that the model treats as a single unit. Common words are often one token. Rarer words, identifiers and punctuation get split into several.

How a few strings might be split into tokens.

A common rule of thumb is that one token is about four characters of English prose [22]. Two caveats matter for programmers:

Code costs more tokens per character than prose. Symbols, brackets and camelCase identifiers split into many small pieces.
Each model family has its own tokenizer, so the same file can have noticeably different counts across models.

Why tokens matter

Tokens are the unit of both cost and capacity.

Cost: you pay per token. Input and output tokens are billed separately, and output tokens cost several times more than input. Pasting a whole file to fix one function wastes input tokens. Asking for a full rewrite where a targeted edit would do wastes output tokens.

Capacity: a fixed budget. The system prompt, tool definitions, messages, files and tool output all share the same context window. The model is also stateless: the whole conversation is re-sent on every request (more on that in section 2). Long sessions fill the window fast.

How the next token is picked

At every step the model assigns a score (a logit) $z_i$ to every token in its vocabulary. The scores are turned into probabilities with a softmax, and one token is sampled from that distribution. The temperature $T$ rescales the scores before the softmax:

$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$

With $T = 1$ you get the model’s own distribution. A lower temperature sharpens it: the top token gets more of the mass. As $T \to 0$ the top token always wins, which is greedy decoding. A higher temperature flattens the distribution, so less likely tokens get picked more often.

The figure takes the prompt “The animal that barks is the ___” and five candidate tokens. It starts from the probabilities at $T = 1$ and applies the formula above to get the other two panels.

Temperature reshapes the same distribution: lower is more focused, higher gives more surprises.

At $T = 0.5$ , dog gets 88% of the mass. At $T = 2$ it drops to 39%, and even car gets picked about one time in nine.

Temperature: match it to the task

The right temperature depends on what you want from the model:

Task	Temperature	Why
Code generation	0	Consistency and correctness
Bug fixing	0	The most probable fix
Code review	0 – 0.3	Surface different issues
Documentation	0.3 – 0.5	Some natural variety
Brainstorming	0.7 – 1.0	Diverse ideas

Coding tools such as Claude Code and Cursor set these values for you. You only deal with them when you call the API directly.

Top-p and top-k: drop the tail, then sample

Temperature reshapes the whole distribution. Two other knobs cut off its unlikely tail before sampling:

Top-p (nucleus sampling) [1] keeps the smallest set of tokens whose probabilities add up to at least $p$ . Because it works on cumulative probability, it adapts to the shape of the distribution: it keeps few tokens when the model is confident and more when it is unsure.
Top-k [2] always keeps the $k$ most likely tokens, however the probability is spread.

The surviving tokens are renormalized and one is sampled. In the figure below, car is never drawn under either rule.

Top-p = 0.9 keeps dog, cat, wolf and fox, since together they reach 0.95. Top-k = 2 keeps only dog and cat.

Model knobs: cheat sheet

Knob	Default	Notes
`temperature`	1.0	0 for code. Change this or `top_p`, not both
`top_p`	1.0 (off)	0.9 – 0.95 when you want controlled variety
`top_k`	off	Rarely exposed by coding tools
`max_tokens`	none	Hard cap on the response length
`stop_sequences`	none	Strings that end generation early

A clearer prompt changes the output far more than any of these settings.

Newer reasoning models drop the sampling knobs altogether: Claude Opus 4.7 and later reject them. You set an effort level instead, which controls how much the model thinks before it answers [3].

2. The context: what does the model see?

The context window is working memory

The context window is everything the model can “see” at once: its instructions, the tool definitions and the whole conversation so far. Whatever is left over is free space [4].

A typical mid-session breakdown of a coding agent’s context. These colours mean the same thing in every figure that follows.

Two parts are sent with every request: the system prompt and the tool definitions. Together they can take close to a third of the window before you type a word. The conversation (your messages, the model’s replies, and above all the tool calls and their results) grows every turn. The remainder is headroom.

The model has no memory between calls

The model keeps nothing between requests. The harness stores the list of messages and re-sends all of it on every request. Each answer is appended and goes out again with the next request.

Every request carries the full history. The system prompt and tool definitions form a stable prefix.

This has three consequences:

Quadratic cost. If every turn re-sends everything before it, the total number of input tokens after $n$ turns grows roughly as $n^2$ . Say the prefix is $P$ tokens and each turn adds $d$ tokens. Then request $k$ carries $P + kd$ tokens, and after $n$ turns you have sent

$\sum_{k=1}^{n} (P + kd) = nP + d\,\frac{n(n+1)}{2}$

tokens in total.

Illustrative numbers: after 40 turns the harness has sent about 23 times more input tokens than the conversation actually contains.

Tool traffic dominates. Tool calls and their results are messages too, and they pile up. A single test run or grep can return thousands of tokens, and those tokens then ride along on every later request.

Keep the prefix stable. Providers cache the unchanged start of a request. With prompt caching, a repeated prefix is billed at a fraction of the normal input price and processed faster [5]. The cache matches byte for byte, so anything that changes the prefix (an edited system prompt, a reordered tool list, a timestamp) invalidates it. Put stable content first and keep it fixed.

Message roles

Every request has two fields that ride along each time, then a list of messages:

Part	What it holds	Note
System prompt	Persistent instructions: persona, rules, constraints	A separate field, not a message. Sent first, every time
Tool metadata	Name, description and JSON schema of every tool	Sent on every request, whether it is used or not
User	Your prompt, and also the results of tool calls	Tool results are observations, not instructions
Assistant	The model’s earlier replies: text and tool calls	Re-sent because the model is stateless

Anatomy of one request

This is what the harness sends to the API [12], slightly trimmed:

{
  "model": "claude-opus-5",
  "system": "You are a coding assistant. Prefer small, tested changes.",
  "tools": [{"name": "read_file",
             "description": "Read a file from the repository.",
             "input_schema": {"type": "object", "required": ["path"],
                              "properties": {"path": {"type": "string"}}}}],
  "messages": [
    {"role": "user", "content": "Why does test_auth fail?"},
    {"role": "assistant", "content": [{"type": "tool_use", "id": "t1",
        "name": "read_file", "input": {"path": "tests/test_auth.py"}}]},
    {"role": "user", "content": [{"type": "tool_result",
        "tool_use_id": "t1", "content": "def test_login(client): ..."}]},
    {"role": "assistant", "content": "The fixture `client` is never..."}
  ]}

Reading it top to bottom: the system prompt, the tool metadata, your prompt, a tool call from the model, the tool result from the harness, and the answer. All of it is re-sent next turn. The tool result arrives as a user message, and the model treats it as something it observed rather than a request from you. That matters when a file or web page contains text that looks like a command.

AGENTS.md: instructions loaded every time

Most harnesses let you keep a project file that is injected into the system prompt on every request. It is the place for what the agent should always know:

# orders-api

## Commands
- Test: uv run pytest -x
- Lint: uv run ruff check --fix

## Conventions
- Python 3.12, type hints everywhere
- Business logic in src/services/
- Never edit files under vendor/

The file name depends on the tool:

Tool	File
Open standard [10]	`AGENTS.md`
Claude Code [11]	`CLAUDE.md`
Cursor	`.cursor/rules/*.mdc`
GitHub Copilot	`.github/copilot-instructions.md`

Treat it like any other config:

Version-control it.
Keep it short. Every request pays for every line.
Put the most important rules first.
Prune stale instructions regularly.

Advertised context is not usable context

Model cards advertise context windows of 128K, 200K or a million tokens. How much of that can a model actually use? The RULER benchmark [6] measures this. It defines a model’s effective length as the longest input at which it still scores above a fixed baseline on retrieval, multi-hop tracing and aggregation tasks. The long-context survey by Liu et al. [7] collects those numbers for a range of models:

Claimed versus effective context length. Log scale, so every gridline is a factor of ten.

Most models hold up for only 12 – 50% of their advertised length, and some for under 2%. These are 2024 models, and newer ones do better. When you budget a long session, use the effective number.

When the window fills up

Quality drops well before the window is full, in three ways:

Lost in the middle. Models use information at the start and end of a long context better than information in the middle [8]. As the window fills, details mid-context get overlooked.
Autocompaction. Near the limit, the harness summarizes older turns. You lose detail, references break, and the agent repeats work it has already done.
Diluted rules. Conflicting instructions compete, and key requirements fade.

A practical threshold: when a session reaches half to 60% of the window, start a fresh one rather than wait for autocompaction.

Compaction: summarize, then drop

Near the limit the harness swaps older turns for a summary. The system prompt, tool definitions and the most recent turns stay as they are.

Before compaction, the window is full of old turns. After, a summary replaces them.

What survives and what is lost:

Survives	Lost
Facts the summary mentions	Exact output, error messages, line numbers
The most recent turns	Earlier reasoning and rejected options
Anything on disk	Constraints you stated only once

The last row of the right column is the dangerous one. If a constraint matters, write it into AGENTS.md or a file on disk, or restate it after compaction.

Cheaper levers than compaction

Compaction is the last resort. There are cheaper ways to keep the window small [9]:

Truncate output. The harness caps long tool results. You can do it too: tail, grep, --quiet.
Clear old results. Keep the tool call but drop its bulky result once it has been used.
Delegate. Run noisy work in a subagent; only its conclusion comes back (see section 4).
Persist to files. Plans and notes on disk survive compaction.
Load fewer tools. Every tool definition is sent every turn. Remove MCP servers you do not use.
Start fresh. The only lossless option: you choose what carries over.

Context hygiene

Do	Don’t
`/clear` between tasks	Paste whole files or logs
Point at files instead of pasting them	Drift across unrelated tasks in one session
Keep `AGENTS.md` short and ordered	Load MCP servers you won’t use
Send noisy work to a subagent	Rely on the model remembering old turns
Filter command output	Wait for autocompaction
Restate key constraints after compaction

3. The loop: how does a model take action?

Tools: the model asks, the harness acts

A model can only produce text. It cannot run a test, read a file or edit code. Tools bridge that gap in three steps [12]:

1. Definition (sent with every request). The harness describes each tool: a name, a description and a JSON schema for its input.

{"name": "run_tests",
 "description": "Run the test suite. Call after every code change.",
 "input_schema": {
   "type": "object",
   "properties": {
     "path": {"type": "string"}}}}

2. Call (model to harness). Instead of plain text, the model emits a structured request to use a tool.

{"type": "tool_use",
 "id": "t1",
 "name": "run_tests",
 "input": {"path": "tests/"}}

3. Result (harness to model). The harness runs the tool and sends the output back, tagged with the same id.

{"type": "tool_result",
 "tool_use_id": "t1",
 "content": "3 passed, 1 failed ..."}

The model never runs code. It emits a structured request; the harness runs it and feeds the result back as text. The description is also a prompt: it tells the model when and how to use the tool. “Call after every code change” in the definition above is an instruction, and the model will follow it.

Typical tools in a coding agent: read, edit, write, grep, glob, bash, web fetch and ask user.

The agent loop

Tool use on its own is one step. Put it in a loop and you get an agent [13].

Tool use plus a loop is agentic coding.

The model thinks about what to do next, acts by calling a tool, and observes the result. If it is not done, it goes round again. That is how it finishes multi-step tasks on its own: search, read, edit, run the tests, and repeat.

How does the loop know when to stop? “Done” means the model replied without requesting a tool. The API signals this with stop_reason = "end_turn" instead of "tool_use".

One turn, step by step

Here is the opening example again, as a sequence of messages between you, the harness, the model and your machine:

One user turn can contain many model calls. The harness sits in the middle of every exchange.

You send one message. The harness sends the system prompt, the tools and the history to the model. The model asks for bash("pytest -x"). The harness checks permissions, runs the command, truncates the output and sends it back. The model asks for an edit, and so on. The loop repeats until the model answers in plain text, and the harness passes that text back to you.

A complete agent in about 20 lines

This is small enough to write out in full. With the Anthropic Python SDK:

import subprocess, anthropic
client = anthropic.Anthropic()
TOOLS = [{"name": "bash", "description": "Run a shell command and return its output.",
          "input_schema": {"type": "object", "required": ["cmd"],
                           "properties": {"cmd": {"type": "string"}}}}]
def run_bash(cmd):                                    # the tool: no checks, no limits
    p = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    return p.stdout + p.stderr
messages = [{"role": "user", "content": input("> ")}]
while True:                                           # the agent loop
    resp = client.messages.create(model="claude-opus-5", max_tokens=16000, tools=TOOLS,
                                  system="You are a coding agent.", messages=messages)
    messages.append({"role": "assistant", "content": resp.content})
    if resp.stop_reason != "tool_use":                # no tool requested: done
        print(next(b.text for b in resp.content if b.type == "text")); break
    results = [{"type": "tool_result", "tool_use_id": b.id, "content": run_bash(b.input["cmd"])}
               for b in resp.content if b.type == "tool_use"]
    messages.append({"role": "user", "content": results})    # results go back as a user turn

Everything from the previous sections is here: the tool definition, the growing messages list that is re-sent every time, the assistant turn with tool calls, the results returned as a user turn, and the stop_reason check that ends the loop.

It works. It will also run rm -rf if the model asks for it, dump a 40 MB log into the context, and hang on npm start, because a dev server never exits.

What a production harness adds

A production harness keeps the same short loop and adds the engineering around it:

Concern	Naive (above)	Real harness
Safety	Runs anything, as you	Allow/deny rules, approval prompts, sandbox
Output size	Everything into the context	Truncated, paginated, offset/limit parameters
Time	Blocks until it finishes	Timeouts, background jobs, cancellation
State	A new shell on every call	Persistent working directory and environment
Errors	An exception kills the loop	Exit code and stderr returned, flagged `is_error`
Description	One sentence	Hundreds of words: when, how, what to avoid
Parallelism	One call at a time	Independent calls run concurrently
Cost	Nothing tracked	Token accounting, prompt caching, compaction

Agent = model + harness

The model is the engine. Everything around it is the harness.

An agent is a model plus a harness. The harness supplies the system prompt (rules, persona, project config), the tools (their definitions and the code that runs them), the loop, context management (compaction, caching), permissions (approvals, sandbox, hooks) and the interface (CLI, IDE, chat, memory). Use the same model with a different harness and you get a different agent. Claude Code, Codex CLI, Cursor and the 20-line script above are all harnesses.

Why the harness matters

Most of what you experience as “the agent” is harness behaviour: which files it reads first, when it asks permission, how it recovers from a failed test, what it forgets.
Agent benchmarks score model plus scaffold. Change the scaffold and the score moves. The SWE-agent paper [14] showed this directly: with the same model, a better agent–computer interface (the tools, their output formats and guardrails) solved far more real GitHub issues.

You rarely get to change the model. The harness you change all the time: config files, permissions, hooks, tools, MCP servers, skills, subagents and worktrees.

4. The harness: where do you have leverage?

The tool landscape

Category	What it is	Examples
API platforms	You build the harness	Anthropic, OpenAI, Gemini
CLI agents	An agent in the terminal	Claude Code, Codex, Gemini CLI
IDE extensions	An agent in the editor	Copilot, Cursor, Windsurf
Chat assistants	Conversation first	Claude, ChatGPT, Gemini

Only the first row leaves the harness to you. The other three ship with one.

Six levers you control

AGENTS.md. What the agent should always know. Loaded on every request (section 2).
Subagents. Isolate noisy work in a separate context window.
Worktrees. Isolate files so several sessions can run in parallel.
Skills. Package a workflow once, and load it only when it is needed.
MCP. Connect new tools and data sources through one protocol.
Hooks. Run deterministic checks at fixed points in the loop.

The examples below use Claude Code, but most harnesses have equivalents.

Subagents: a fresh context for noisy work

A subagent is a specialized assistant with its own context window and a restricted tool set [15]. The main agent hands it a task. The subagent does the noisy work (reading dozens of files, running searches) in its own window, and only its conclusion returns to the main conversation.

The main agent sees the task and the summary. The subagent’s tool traffic never enters the main context.

This gives you three things:

Context isolation. Verbose output stays out of the main conversation.
Parallelism. Several subagents can run at once.
Cost control. Simple tasks can go to cheaper, faster models.

Claude Code ships with built-in types. Explore and Plan are read-only; General has full access.

Worktrees: parallel sessions, isolated files

A subagent isolates context; a worktree isolates files. A git worktree is an extra working directory attached to the same repository [16].

One repository, three working directories, three branches.

Each worktree has its own files, branch and working state.
Commit history and remotes are shared.
You can run parallel features, experiments or isolated agents without them overwriting each other’s changes.

# plain git
git worktree add ../app-auth -b feature-auth

# Claude Code shortcut
claude --worktree feature-auth

See the Claude Code docs for the full workflow [17].

Skills: reusable workflows, loaded on demand

A skill is a folder with a SKILL.md file: instructions, plus optional scripts and reference files [18].

---
name: explain-code
description: Explains code with diagrams
user-invocable: true
allowed-tools: Read, Grep, Glob
---
When explaining code:
1. Draw a diagram: ASCII art
2. Walk through: step-by-step
3. Highlight a gotcha

Skills are cheap until used. Only the name and description sit in the context. The body is loaded when the skill runs. Compare that with AGENTS.md, which is paid for on every request. You can invoke a skill yourself as a slash command:

/explain-code src/auth/middleware.ts

or let the model pick it based on its description.

MCP: the Model Context Protocol

The Model Context Protocol (MCP) [19] is an open standard for connecting models to tools and data. It is often called “USB for AI”.

Many clients, many servers, one protocol in between.

Write a server once and any client can use it. Adding a server to Claude Code is one command [20]:

claude mcp add context7 \
  --transport http --scope user \
  https://mcp.context7.com/mcp

A team can share its setup in a .mcp.json file at the repository root.

The catch: every server adds its tool definitions to every request. Recall the context breakdown from section 2: tools can already be a fifth of the window. Before adding a server, make sure the capability is worth the context it costs.

Hooks: rules that always run

Hooks are shell commands that run at fixed points in the loop [21]. The model can overlook an instruction in AGENTS.md. A hook runs every time its event fires.

Hook events sit at fixed points of the loop. Your command runs there every time.

PreToolUse. Runs before a tool executes. Use it to block dangerous commands.
PostToolUse. Runs after a tool executes. Use it to auto-format edited files.
Notification. Alerts you when the agent needs input.
Stop. Runs when the agent wants to finish. Use it to check that the task is really done, for example that the tests pass.

Here is a PostToolUse hook that runs ruff format on every file the agent edits or writes. The hook receives the tool call as JSON on stdin:

{"hooks": {"PostToolUse": [{"matcher": "Edit|Write",
  "hooks": [{"type": "command",
    "command": "jq -r '.tool_input.file_path' | xargs ruff format"}]}]}}

Put it in .claude/settings.json, and formatting stops depending on whether the model remembered to do it.

5. Wrap-up: what to take home

Context is the scarce resource. Start fresh sessions, watch your usage, and put the key instructions first.
Be specific. Name the file and the function, define what “done” means, and state the constraints.
The loop does the work. Think, act, observe, until the model stops asking for tools.
Agent = model + harness. The loop is 20 lines. Most of the work sits in the tools, context handling and permissions around it.
You own the result. Commit often, review every diff, and run the tests.

Take-home exercise: FizzBuzz, the agentic way

Apply these ideas to a simple problem, then evaluate the result:

Try a vague prompt first, then a structured one, and compare.
Watch the context usage as you go (/context in Claude Code).
Review the diff and run the tests yourself.

The course notes, with more detail on each topic, are at anoopkcn.github.io/ai-assisted-coding.

References

A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi. The Curious Case of Neural Text Degeneration. ICLR 2020. arXiv:1904.09751
A. Fan, M. Lewis, Y. Dauphin. Hierarchical Neural Story Generation. ACL 2018. arXiv:1805.04833
Anthropic. Effort. Claude API documentation. docs.claude.com
Anthropic. Context windows. Claude API documentation. docs.claude.com
Anthropic. Prompt caching. Claude API documentation. docs.claude.com
C.-P. Hsieh et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? COLM 2024. arXiv:2404.06654
J. Liu et al. A Comprehensive Survey on Long Context Language Modeling. 2025, §7.1.1. arXiv:2503.17407
N. F. Liu et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. arXiv:2307.03172
Anthropic. Effective context engineering for AI agents. 2025. anthropic.com
AGENTS.md, an open format for guiding coding agents. agents.md
Anthropic. Manage Claude’s memory (CLAUDE.md). Claude Code documentation. code.claude.com
Anthropic. Tool use with Claude. Claude API documentation. docs.claude.com
E. Schluntz, B. Zhang. Building effective agents. Anthropic, 2024. anthropic.com
J. Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. arXiv:2405.15793
Anthropic. Subagents. Claude Code documentation. code.claude.com
git-worktree: Manage multiple working trees. Git documentation. git-scm.com
Anthropic. Common workflows: run parallel sessions with git worktrees. Claude Code documentation. code.claude.com
Anthropic. Agent Skills. Claude Code documentation. code.claude.com
Model Context Protocol. Specification and documentation. modelcontextprotocol.io
Anthropic. Connect Claude Code to tools via MCP. Claude Code documentation. code.claude.com
Anthropic. Hooks reference. Claude Code documentation. code.claude.com
OpenAI. What are tokens and how to count them? OpenAI Help Center. help.openai.com

Tombili of Istanbul: Relax and Enjoy for a Moment

Fri, 11 Apr 2025 10:46:29 +0000

Tombili of Istanbul: Relax and Enjoy for a Moment

Tombili of Istanbul

CUDA Programming Model and CUDA Fortran

Wed, 18 Apr 2018 00:00:00 +0000

CUDA Programming Model and CUDA Fortran

Note, September 2026: this post is from 2018. Since then the PGI compilers have become part of the NVIDIA HPC SDK, and pgf90 is now nvfortran. With nvfortran, -cuda enables CUDA Fortran and -gpu=cc60 selects the compute capability. The commands below keep the 2018 names.

CUDA-enabled GPUs are capable of running tens of thousands of threads concurrently. This functionality is possible due to a large number of processor cores available on each GPU device. They are grouped into multiprocessors. The design of the CUDA programming model is done such that it can utilise the massive parallelism capability of the GPUs. A subroutine or a kernel is launched with a grid of threads grouped into thread blocks. These thread blocks can operate independently of one another thereby allowing scalability of a program. In addition to that data can be shared between these threads enabling more granular data parallelism.

The available multiprocessors can schedule these blocks of threads in any order allowing the program to be executed on any device with any number of multiprocessors. This scheduling is performed automatically so that one can concentrate on the partition of the program into fractions that can be run and parallelised on a block of threads.

CUDA programming model allows utilising both CPU and GPU to perform computations. This hybrid computing capability can be used to achieve massive performance improvements. A typical sequence of operations for a simple CUDA Fortran code are:

Declare and allocate host (CPU) and device (GPU) memory
Initialize host data
Transfer data from the host to the device
Execute one or more kernels which perform operations on the device data
Transfer results from the device to the host

There are a number of good programming practices to be observed when moving data back and forth from host to device and vice-versa. Among those the most noticeable being the bandwidth of PCI and bandwidth between device’s memory and GPU, where the former is an order of magnitude lower than that of the latter. From a performance point of view, the less number of data transfers between the host and the device the better will be the performance.

The allocate() command and the assignment operators are overloaded in the CUDA Fortran model so that it can also allow allocation of the device memory (using device attribute) and data transfer between the host and device memory spaces. The Fortran2003 source allocation construct, allocate(A, source=B), for cloning B into A is also extended. In CUDA Fortran, if the A array is defined as a device array and B is defined as host array, then contents of B will be copied over the PCI bus to A. These methods of data transfer are all blocking transfers, in that control is not returned to the CPU thread until the transfer is complete. This prevents the possibility of overlapping data transfers with computation on both the host and device. For concurrent computation one might employ CUDA API function cudaMemcpyAsync() to perform asynchronous data transfers.

Kernels, or subroutines are typically invoked in host code just as any subroutine is called, but since the kernel code is executed by many threads in parallel an additional execution configuration has to be provided to indicate thread specifications. Data elements are mapped to threads using the automatically defined variables threadIdx, blockIdx, blockDim, and gridDim.

One of the functionalities provided by CUDA Fortran is that it can automatically generate and invoke kernel code from a region of host code containing tightly nested loops. Such type of kernels are referred to as CUF kernels. A simple example of a CUF kernel is:

!$cuf kernel do <<<*,*>>>
do i=1, n
 ! a_d is a device array, b is a host scalar
 a_d(i) = a_d(i) + b
end do

In the above code wild-cards are used to make the runtime system determine the execution configuration. Even though the scalar b is a host variable it is passed as a kernel argument by value upon the execution of the CUF kernel.

In the following code subroutine add_scalar is making use of the module array_def which defines the array a(n)

module array_def
 real :: a(n)
end module array_def
!
! Following subroutine make use of the module array_def
subroutine add_scalar
 use array_def, only: a
 do i=1, n
  a(i) = a(i) + b
 end do
end subroutine add_scalar

Converting this code to a GPU friendly one can be done as follows:

module array_def
 real :: a(n)
 real,device :: a_d(n)
end module array_def
!
subroutine add_scalar
#ifdef USE_GPU
 use array_def, only: a => a_d
#else
 use array_def, only: a
#endif
!$cuf kernel do <<<*,*>>>
 do i=1, n
  a(i) = a(i) + b
 end do
end subroutine add_scalar

When compiling without the preprocessor directive USE_GPU, the code will behave the same as the CPU version, ignoring the CUF kernel execution configuration. When GPU is enabled the device allocated array a_d(n) is used instead, for a calculation respecting the CUF kernel.

If the arrays used in the subroutine are explicitly passed then one can make use of the device attribute to achieve the same effect as above. The following code illustrates the same.

subroutine add_scalar(a,n)
 real:: a(n)
#ifdef USE_GPU
 attributes(device) :: a
#endif
!$cuf kernel do <<<*,*>>>
 do i=1, n
  a(i) = a(i) + b
 end do
end subroutine add_scalar

Compiling and Running the Code

Any code in a file with a .cuf or .CUF extension is compiled with CUDA Fortran automatically enabled. Otherwise, the compiler flag -Mcuda can be used to compile CUDA Fortran code with other extensions. If above code is in a file named cufortran.cuf, compilation and execution of the code is as simple as:

pgf90 -o cufortran cufortran.cuf
./cufortran

For files that are intended to compile both on CPU and GPU, as explained in this post, one can compile the code using the following sequence.

pgf90 -Mcuda=cc60 -O3 -Mpreprocess -DUSE_GPU cufortran.F90

The preprocessor directive -Mpreprocess -DUSE_GPU will activate the CUDA parts of the code when used. The value of the -Mcuda=cc60 flag will be inferred by the GPU card one is using: cc60 is for NVIDIA P100 cards and cc35 is for K80 cards and so on.

A community edition of the PGI compiler (>= version 16.10) is available at the NVIDIA website, where CUDA Fortran is implemented.

References

Using crossref for building a bib

Fri, 24 Nov 2017 00:00:00 +0000

Using crossref for building a bib

tl;dr

This is based on a one-liner post. Provided a DOI of a paper or an article, the following will use the crossref API [1] and display the information regarding the document such as author, year, publication, etc.

#DOI=DOI of a paper
DOI='DOI';
curl -s "https://api.crossref.org/works/${DOI}/transform/application/x-bibtex"

The following is a bash function which makes use of this one-liner to do slightly complicated tasks such as appending the crossref info of a document to a file, selectively displaying the information when not given a file, etc.

Bash Function

The file variable is initially empty, this is to store the file name the user would provide upon calling the function doi2bib with the flag -f. Likewise info is an empty string which would hold the information tag such as author, year, publisher, volume etc., provided by the user with the flag -i. The flags -f and -i are mutually exclusive in the following script. Flag -d is mandatory since this specifies the DOI of the document in question.

doi2bib() {
  local doi="" info="" file=""
  local api="https://api.crossref.org/works" trans="transform/application/x-bibtex"
  while [ $# -gt 0 ]; do
    case "$1" in
      -h|--help)
        echo "usage: doi2bib -d DOI [-i FIELD | -f FILE]"
        return 0 ;;
      -d|--doi)  doi=${2-};  shift ;;
      -i|--info) info=${2-}; shift ;;
      -f|--file) file=${2-}; shift ;;
      *) echo "doi2bib: unknown option $1" >&2; return 1 ;;
    esac
    [ $# -gt 0 ] && shift
  done
  if [ -z "$doi" ]; then
    echo "doi2bib: no DOI given, use -d DOI" >&2; return 1
  fi

  local bib
  bib=$(curl -sfL "$api/$doi/$trans") || {
    echo "doi2bib: crossref has no BibTeX for $doi" >&2; return 1
  }

  if [ -n "$file" ]; then
    printf '\n%s\n' "$bib" >> "$file"
  elif [ -n "$info" ]; then
    grep -oE "$info=(\{[^}]*\}|[^,} ]*)" <<< "$bib"
  else
    printf '%s\n' "$bib"
  fi
}

A few typical examples of the usage of the above script are given as follows.

To show the full info on the terminal:

doi2bib -d https://doi.org/10.1016/j.jssc.2016.08.030

To show the selective info on the terminal:

doi2bib -d https://doi.org/10.1016/j.jssc.2016.08.030 -i author

To append the full information in the bibtex format to a file named mypaper.bib:

doi2bib -d https://doi.org/10.1016/j.jssc.2016.08.030 -f mypaper.bib

References

The online service of crossref