Configure

Configure PQS data sources, filters, and encoding options.

PQS ascertains its configuration from the following sources, in order of priority:

• HOCON configuration files (--config argument)

• command-line arguments

• Java system properties (-D arguments to java command)

• environment variables

Consult the command ./scribe.jar pipeline --help-verbose for further information on individual configuration items, and the conventions used to specify them in the above forms.

PQS sets its configuration at startup. It does not perform dynamic configuration updates, so making a configuration change (such as adding a new party, or new interface to filters) requires a restart.

Deploying new Daml packages (DARs) to the Participant Node does not require restarting PQS. When PQS encounters an unknown package while processing events, it fetches the missing packages from the Participant Node. This could pause ingestion for up to tens of seconds. See Dynamic Daml package reload for details.

PQS will not go back in time and recover history, but only move forward by consuming new transactions it has not previously seen. It is important that scope only be expanded when it is known to have no prior history, at the point in time that PQS was stopped. Otherwise, a re-seed operation will be required to reinitialize from an empty datastore.

The following command connects to a non-auth ledger and replicates the latest state of the ledger (excluding prior-history) from the perspective of the supplied Daml party. It uses the ledger source and supplied database:

$ ./scribe.jar pipeline ledger postgres-document \
     --pipeline-filter-parties "Alice::* | Bob::*" \
     --pipeline-ledger-start Latest \
     --source-ledger-host canton-host \
     --source-ledger-port 6865 \
     --target-postgres-host postgres-host \
     --target-postgres-port 5432 \
     --target-postgres-database postgres

Consult Configuration options how to configure each command.

Transactions data source

To understand how PQS stores data, you need to understand the Ledger Model. In simple terms, the Daml ledger is composed of a sequence of transactions, which contain events. Events can be:

• Create: creation of contracts / interface views / divulgences

• Exercise: of a choice of contracts / interface views

• Archive: end of the lifetime of contracts / interface views

When defining the scope of ledger data being stored, it is important to understand the implications of the data source and the filters applied. The data source and filters determine the data that is available to the SQL functions (see SQL API), and this cannot be changed. Since a change in scope will result in a change to the breadth of data being stored, a re-seed is required to widen or narrow the scope of the data. The only exception to this is where you widen the scope into an area, for example new templates, new parties, that you know has no historical data, in which case a re-seed is not required. Or, operators may also use the reset function to roll-back the datastore to a prior state where this was true.

PQS can run in two modes as specified by the --pipeline-datasource configuration. The following table shows the differences between the two modes, in terms of data availability via the respective SQL functions:

Data / Mode	TransactionStream	TransactionTreeStream
creates() of contracts	✓	✓
exercises() on contracts	✗	✓
archives() of contracts	✓	✓
creates() of interfaces	✓	✓
exercises() on interfaces	✗	✓
archives() of interfaces	✓	✓
creates() of divulgences	✗	✓
Transient (create-archive in a transaction)	✗	✓
Stakeholders & witnesses	✓	✓
Choice controllers	✗	✓
Note
Default	✓	✗
Data size	Smaller	Larger

Contract filtering

--pipeline-filter-contracts specifies a filter expression to determine which the Daml templates, interface views and choices to include. A filter expression is a simple wildcard inclusion (*) with basic boolean logic (& ! | ( )), where whitespace is ignored. For example:

• *: everything (default)

• pkg:*: everything in this package

• pkg@=1: everything in this package that has major version 1.x.x

• pkg@>=1.2.3: everything in this package starting with version 1.2.3 inclusively

• foo@1.2.3|bar@3.2.1: everything in pinpointed packages foo at version 1.2.3 and package bar at version 3.2.1

• pkg:a.b.c.Bar: just this one fully qualified name for template Bar

• a.b.c.*: all members of the a.b.c namespace

• * & !pkg:a.b.c.Bar: everything except this one fully qualified name

• (a.b.c.Foo | a.b.c.Bar): these two fully qualified names

• (a.b.c.* & !(a.b.c.Foo | a.b.c.Bar) | g.e.f.Baz): everything in a.b.c except for Foo and Bar, and also include g.e.f.Baz

• a.b.c.Foo & a.b.c.Bar: error (the identifier can’t be both)

There are further conditions placed upon the filtering of templates and interfaces to avoid potential ambiguity. It is required to include any filter for:

• All interface views of included templates

• All templates of included interface views

Party filtering

Similarly, the --pipeline-filter-parties option specifies a filter expression to determine which parties to supply data for. For example:

• *: everything (default)

• Alice::* | Bob::*: any party with an Alice or Bob hint

• Alice::122055fc4b190e3ff438587b699495a4b6388e911e2305f7e013af160f49a76080ab: just this one party

• * & !Alice::*: all parties except those with an Alice hint

• Alice* | Bob* | (Charlie* & !(Charlie3::*)): Alice and Bob parties, as well as Charlie except Charlie3

When the ledger requires authentication, this filter applies within the scope of parties for which PQS’ Ledger API user has access. Naturally, the --pipeline-filter-parties cannot be used to access data for parties for which the user is not authorized.

Explicit contract disclosure

--pipeline-filter-metadata specifies an inclusion filter expression to determine the Daml templates and interface views to capture metadata for. Same syntax as section above applies. Captured data will be available in the metadata column (bytea PostgreSQL type) in the query functions output. This column stores the contents of created_event_blob (see How do stakeholders disclose contracts to submitters?) from the respective event.

JSON encoding configuration

By default PQS stores the payloads in JSON format using strings for numeric types. This is the default because numbers in JSON are backed by double-precision floating point numbers, which can lead to loss of precision for large integers. If you want to store numbers as integers and can tolerate loss of precision in numbers, you can use the --target-encoding-numericasstring and --target-encoding-int64asstring options to change the default behavior.

Another option allows to store nullable fields as JSON nulls instead of omitting nullable fields from the records. PQS stores nullable fields as JSON nulls by default, but you can change this behavior using the --target-encoding-excludenulls option.