v0.39.0

DynamicPPL v0.39.0

Diff since v0.38.10

Breaking changes

Fast Log Density Functions

This version provides a reimplementation of LogDensityFunction that provides performance improvements on the order of 2–10× for both model evaluation as well as automatic differentiation.
Exact speedups depend on the model size: larger models have less significant speedups because the bulk of the work is done in calls to logpdf.

For more information about how this is accomplished, please see #1113 as well as the src/logdensityfunction.jl file, which contains extensive comments.

As a result of this change, LogDensityFunction no longer stores a VarInfo inside it.
In general, if ldf is a LogDensityFunction, it is now only valid to access ldf.model and ldf.adtype.
If you were previously relying on this behaviour, you will need to store a VarInfo separately.

Threadsafe evaluation

DynamicPPL models have traditionally supported running some probabilistic statements (e.g. tilde-statements, or @addlogprob!) in parallel.
Prior to DynamicPPL 0.39, thread safety for such models used to be enabled by default if Julia was launched with more than one thread.

In DynamicPPL 0.39, thread-safe evaluation is now disabled by default.
If you need it (see below for more discussion of when you do need it), you must now manually mark it as so, using:

@model f() = ...
model = f()
model = setthreadsafe(model, true)

The problem with the previous on-by-default is that it can sacrifice a huge amount of performance when thread safety is not needed.
This is especially true when running Julia in a notebook, where multiple threads are often enabled by default.
Furthermore, it is not actually the correct approach: just because Julia has multiple threads does not mean that a particular model actually requires threadsafe evaluation.

A model requires threadsafe evaluation if, and only if, the VarInfo object used inside the model is manipulated in parallel.
This can occur if any of the following are inside Threads.@threads or other concurrency functions / macros:

• tilde-statements
• calls to @addlogprob!
• any direct manipulation of the special __varinfo__ variable

If you have none of these inside threaded blocks, then you do not need to mark your model as threadsafe.
Notably, the following do not require threadsafe evaluation:

• Using threading for any computation that does not involve VarInfo. For example, you can calculate a log-probability in parallel, and then add it using @addlogprob! outside of the threaded block. This does not require threadsafe evaluation.
• Sampling with AbstractMCMC.MCMCThreads().

For more information about threadsafe evaluation, please see the Turing docs.

When threadsafe evaluation is enabled for a model, an internal flag is set on the model.
The value of this flag can be queried using DynamicPPL.requires_threadsafe(model), which returns a boolean.
This function is newly exported in this version of DynamicPPL.

Parent and leaf contexts

The DynamicPPL.NodeTrait function has been removed.
Instead of implementing this, parent contexts should subtype DynamicPPL.AbstractParentContext.
This is an abstract type which requires you to overload two functions, DynamicPPL.childcontext and DynamicPPL.setchildcontext.

There should generally be few reasons to define your own parent contexts (the only one we are aware of, outside of DynamicPPL itself, is Turing.Inference.GibbsContext), so this change should not really affect users.

Leaf contexts require no changes, apart from a removal of the NodeTrait function.

ConditionContext and PrefixContext are no longer exported.
You should not need to use these directly, please use AbstractPPL.condition and DynamicPPL.prefix instead.

ParamsWithStats

In the 'stats' part of DynamicPPL.ParamsWithStats, the log-joint is now consistently represented with the key logjoint instead of lp.

Miscellaneous

Removed the method returned(::Model, values, keys); please use returned(::Model, ::AbstractDict{<:VarName}) instead.

The unexported functions supports_varname_indexing(chain), getindex_varname(chain), and varnames(chain) have been removed.

The method DynamicPPL.init (for implementing AbstractInitStrategy) now has a different signature: it must return a tuple of the generated value, plus a transform function that maps it back to unlinked space.
This is a generalisation of the previous behaviour, where init would always return an unlinked value (in effect forcing the transform to be the identity function).

The family of functions returned(model, chain), along with the same signatures of pointwise_logdensities, logjoint, loglikelihood, and logprior, have been changed such that if the chain does not contain all variables in the model, an error is thrown.
Previously the behaviour would have been to sample missing variables.

Merged pull requests:

• Standardise :lp -> :logjoint (#1161) (@penelopeysm)

v0.38.10

DynamicPPL v0.38.10

Diff since v0.38.9

returned(model, chain) and pointwise_logdensities(model, chain) will now error if a value for a random variable cannot be found in the chain.
(Previously, they would instead resample such variables, which could lead to silent mistakes.)

If you encounter this error and it is accompanied by a warning about hasvalue not being implemented, you should be able to fix this by using FlexiChains instead of MCMCChains.
(Alternatively, implementations of hasvalue for unsupported distributions are more than welcome; these must be provided in the Distributions extension of AbstractPPL.jl.)

Merged pull requests:

• Fast InitContext (#1125) (@penelopeysm)
• Allow generation of ParamsWithStats from FastLDF plus parameters, and also bundle_samples (#1129) (@penelopeysm)
• Implement predict, returned, logjoint, ... with OnlyAccsVarInfo (#1130) (@penelopeysm)
• FastLDF / InitContext unified (#1132) (@penelopeysm)
• Remove NodeTrait (#1133) (@penelopeysm)
• Re-enable Mooncake in tests (#1135) (@penelopeysm)
• CompatHelper: add new compat entry for Mooncake at version 0.4 for package test, (keep existing compat) (#1136) (@github-actions[bot])
• Make FastLDF the default (#1139) (@penelopeysm)
• Improve FastLDF type stability when all parameters are linked or unlinked (#1141) (@penelopeysm)
• Comparative benchmarks (#1143) (@penelopeysm)
• Improvements to benchmark outputs (#1146) (@penelopeysm)
• Make threadsafe evaluation opt-in (#1151) (@penelopeysm)
• disable fallback for returned and pointwise_logdensities (#1159) (@penelopeysm)
• Bump actions/checkout from 5 to 6 (#1160) (@dependabot[bot])

Closed issues:

• Depreciate VarInfo in favour of SimpleVarInfo (#416)
• Remove NodeTrait (#1052)
• Implement ParamsWithStats(::FastLDF, ::AbstractVector{<:Real}) (#1119)
• Path to FastLDF (#1120)
• Implement allocation tests for FastLDF (#1126)
• Model functions look different inside testsets (#1127)
• LogDensityFunction with dimension unpreserving transformations (#1149)

v0.38.9

DynamicPPL v0.38.9

Diff since v0.38.8

Remove warning when using Enzyme as the AD backend.

Merged pull requests:

• CompatHelper: add new compat entry for AbstractMCMC at version 5 for package docs, (keep existing compat) (#1114) (@github-actions[bot])
• Add support for AutoEnzyme in logdensityfunction (#1117) (@wsmoses)

v0.38.8

DynamicPPL v0.38.8

Diff since v0.38.7

Added a new exported struct, DynamicPPL.ParamsWithStats.

This can broadly be used to represent the output of a model: it consists of an OrderedDict of VarName parameters and their values, along with a stats NamedTuple which can hold arbitrary data, such as (but not limited to) log-probabilities.

Implemented the functions AbstractMCMC.to_samples and AbstractMCMC.from_samples, which convert between an MCMCChains.Chains object and a matrix of DynamicPPL.ParamsWithStats objects.

Merged pull requests:

• Implement AbstractMCMC.{to,from}_samples (again) (#1112) (@penelopeysm)

v0.38.7

DynamicPPL v0.38.7

Diff since v0.38.6

Made a small tweak to DynamicPPL's compiler output to avoid potential undefined variables when resuming model functions midway through (e.g. with Libtask in Turing's SMC/PG samplers).

Merged pull requests:

• Fix potential undefined variable in model compiler output (#1110) (@penelopeysm)

Closed issues:

• Rename the only_ddpl keyword arg of determine_suitable_varinfo (#1104)

v0.38.6

DynamicPPL v0.38.6

Diff since v0.38.5

Renamed keyword argument only_ddpl to only_dppl for Experimental.determine_suitable_varinfo.

v0.38.5

DynamicPPL v0.38.5

Diff since v0.38.4

Improve performance of VarNamedVector, mostly by changing how it handles contiguification.

Merged pull requests:

• VNV type stability improvements and tests (#1102) (@mhauru)
• Bump gha versions (#1109) (@penelopeysm)
• Bump patch version to 0.38.5 (#1111) (@mhauru)

v0.38.4

DynamicPPL v0.38.4

Diff since v0.38.3

Improve performance of VarNamedVector. It should now be very nearly on par with Metadata for all models we've benchmarked on.

Merged pull requests:

• Improvements to VarNamedVector (#1098) (@mhauru)
• Print out intermediate benchmark results (#1100) (@mhauru)

v0.38.3

DynamicPPL v0.38.3

Diff since v0.38.2

Add an implementation of returned(::Model, ::AbstractDict{<:VarName}).
Please note that although the NamedTuple method still exists, we generally recommend using Dict, as NamedTuples cannot correctly represent variables with indices / fields on the left-hand side of tildes, like x[1] or x.a.

The generic method returned(::Model, values, keys) is deprecated and will be removed in the next minor version.

Merged pull requests:

• Add to_chains and from_chains function (#1087) (@penelopeysm)
• Revert "Add to_chains and from_chains function (#1087)" (#1093) (@penelopeysm)
• tidy integration test workflow (#1094) (@penelopeysm)
• Implement returned for AbstractDict; deprecate {values, keys} method (#1096) (@penelopeysm)

Closed issues:

• Unable to compute sparse Hessians with ThreadSafeVarInfo (#1081)
• returned doesn't have a method for Dict (#1095)

v0.38.2

DynamicPPL v0.38.2

Diff since v0.38.1

Added a compatibility entry for JET@0.11.

Merged pull requests:

• JET@0.11 compat (#1089) (@penelopeysm)

Closed issues:

• Add a logdensity specialization that takes NamedTuples (#671)
• Add support for ProductNamedTupleDistribution (#801)
• Enzyme doesn't like accumulators (#947)
• rogue idea: make SamplingContext a leaf context + make a new InitialisationContext (#955)
• computation time of returned scales with number of model parameters (#1019)