# Expression language

The language of every `_x` field, of `state.derived`, of `functions` (`let`, `x`), of outcome
`derived_x`, and of `let` in tactics. It is a safe subset of Python expressions: if you can read
Python, you can read it.

An expression has no side effects: it reads the state and returns one value. Changing the state
is done by ops (see [reference.md](https://tabletwin.com/docs/reference/#11-ops)).

## 1. Values

| Kind | Example | Note |
|---|---|---|
| none | `None` | `null` in YAML |
| truth value | `True`, `False` | |
| number | `3`, `-2`, `2.5`, `1e3`, `0x1f`, `0o17`, `0b101`, `1_000` | One number type for integers and decimals |
| text | `'abc'`, `"it's"`, `r'a\n'`, `'''long'''` | |
| list | `[1, 2]` | |
| tuple | `(1, 'x')`, `1,`, `()` | Never equal to a list |
| dict | `{'power': 3}` | **Keys are text** |
| set | `{1, 2}`, `set()` | |
| generator | `(x for x in L)`, `zip(...)`, `enumerate(...)` | Lazy, usable once |
| function | `len`, `modified`, board and own functions | |

`True` and `False` count as 1 and 0 (`True + True == 2`). `1 == 1.0 == True`, and these three
are the same key in a set.

## 2. Syntax

**Allowed:**
- literals as above; adjacent text literals are joined (`'a' 'b'` is `'ab'`);
- lists, tuples, dicts (also `{**d, 'x': 1}`) and sets;
- arithmetic `+ - * / // % **`, unary `-` and `+`;
- comparisons `== != < <= > >=`, also chained (`0 < x <= 3`);
- `in`, `not in`, `is`, `is not`;
- `and`, `or`, `not`, and `x if c else y`;
- indexing `a[i]` and slicing `a[i:j:k]`;
- calls with keyword arguments (`max(L, key=len)`);
- the six methods in [§5](#5-methods);
- comprehensions `[...]`, `{...}`, `{k: v ...}`, with several `for` and `if` clauses and tuple
  unpacking (`for k, v in d.items()`);
- generator expressions, also as the only argument without extra parentheses
  (`sum(x for x in L)`);
- line breaks anywhere (they count as spaces); names with non-ASCII letters.

**Not allowed** (a syntax error naming the construct; the validator reports it, and the engine
when it first evaluates the expression):
- assignment: `=`, `:=`, `+=` and the other augmented forms;
- `lambda`, `yield`, `await`, `async`, and statement keywords (`def`, `import`, …);
- names that start with `_` (also as keyword argument or loop variable);
- attributes other than the six methods (`x.append` is an error);
- bit operators `| & ^ ~ << >>` and `@`;
- `*` and `**` unpacking in calls and lists (`**` inside a dict literal is allowed);
- f-strings, bytes (`b''`), complex numbers (`1j`), `...`;
- comments (`#`), `;`, a lone `!`;
- more than one expression;
- a positional argument after a keyword argument; a generator argument next to other arguments
  without its own parentheses (`sum(x for x in L, 0)`).

Text escapes: `\n \t \r \0 \\ \' \"`, `\xhh`, `\uhhhh`, `\Uhhhhhhhh`, backslash-newline; any other
backslash stays as is. In `r'...'` nothing is escaped.

Since `|` is not allowed, a set union is written `set(list(a) + list(b))`.

## 3. Meaning

- `and`/`or` return an operand, not `True`/`False`: `0 or 'x'` is `'x'`. They evaluate no more
  than needed: `1 and 0 and 1/0` is `0`.
- A comparison chain stops at the first that fails: `1 < 0 < 1/0` is `False`.
- `None`, `False`, `0`, and empty text, lists, tuples, dicts and sets are false; everything else
  is true.
- **Arithmetic:**
  - `/` always divides exactly (`7/2` is `3.5`);
  - `//` rounds down (`-7 // 2` is `-4`);
  - `%` takes the sign of the divisor (`-7 % 3` is `2`);
  - dividing by zero is a `ZeroDivisionError`; so is `0 ** -1`.
- `+` joins texts, lists, or tuples of the same kind. `*` repeats them with an integer.
- **Sets:** `-` gives the difference; `<=`, `<`, `>=`, `>` test subsets. Set elements must be
  hashable: `None`, booleans, numbers, text, tuples of those. A list or dict in a set is a
  `TypeError`.
- **Ordering** (`<` etc., `sorted`, `min`, `max`): numbers with numbers, text with text, or lists
  (or tuples) of the same kind, element by element. Anything else is a `TypeError`.
- **Equality** is deep: lists, tuples, dicts and sets compare by content. A list never equals a
  tuple.
- `is` compares identity; use it only with `None` (`x is None`).
- `in`: element of a list, tuple or set; key of a dict; substring of a text.
- **Indexing:**
  - negative indexes count from the end;
  - a slice out of range gives fewer elements;
  - text is indexed per character (code point);
  - a missing key is a `KeyError`, an index out of range an `IndexError`.
- Iterating over a dict gives its keys.
- **Comprehensions** have their own scope: loop variables do not leak out, and the first `in`
  value is computed outside that scope.
- Generators are lazy: `any(1/x for x in [1, 0])` is `True` without dividing by zero. A
  generator can be consumed only once and has no `len`.

## 4. Built-in functions

These are all the built-in functions; any other name must come from the model.

| Function | Meaning |
|---|---|
| `len(x)` | Length of text, list, tuple, dict or set |
| `any(it)`, `all(it)` | Lazy; stop at the first value that decides |
| `min(...)`, `max(...)` | From one iterable or several arguments; `key=f`, `default=v` (for an empty iterable). On ties the first wins |
| `sum(it, start=0)` | Sum (also lists with `start=[]`); not for text. Numbers are summed with compensation like Python ≥ 3.12: `sum([1, 0.1, 0.2]) == 1.3` |
| `range(n)`, `range(a, b[, step])` | List of integers (at most 10,000,000) |
| `abs(x)` | Absolute value |
| `round(x)`, `round(x, n)` | Half to even (`round(2.5)` is `2`), exact on the decimal value (`round(2.675, 2)` is `2.67`) |
| `sorted(it, key=f, reverse=False)` | Stable; `reverse` keeps equal values in their order |
| `list(x)`, `tuple(x)`, `set(x)` | Convert any iterable (no argument: empty) |
| `dict(x)`, `dict(k=v)` | From a dict, from pairs with text keys, and/or keyword arguments |
| `enumerate(it, start=0)`, `zip(a, b, ...)` | Lazy; yield tuples |
| `int(x)` | Truncates toward zero; text must be an integer (`int('3.5')` is a `ValueError`) |
| `float(x)` | Number or text (`'inf'`, `'nan'` allowed) |
| `str(x)` | Text form (Python notation; see [§10](#10-differences-from-python)) |
| `bool(x)` | Truth value |
| `sigmoid(x)` or `σ(x)` | 1 / (1 + e^−x) |
| `exp(x)`, `sqrt(x)`, `log(x[, base])` | Maths; `log(0)` and `sqrt(-1)` are a `ValueError` (`math domain error`) |

Functions the model or engine adds:

| Function | When | Meaning |
|---|---|---|
| `modified(stat, value, **context)` | Always (not in `init_x`) | `value` after all active modifiers for `stat`; the keyword arguments are names in the modifiers' `if_x` and `change_x`. See [reference.md](https://tabletwin.com/docs/reference/#15-modifiers) |
| `cells`, `directions`, `xy`, `cell`, `step`, `leap`, `ray`, `neighbours`, `distance`, `path`, `path_cost`, `reachable`, `line_of_sight`, `shape` | When the model has `boards` | Board geometry, board name first. See [board-geometry.md](https://tabletwin.com/docs/board-geometry/) |
| Own functions | From `functions` | Called by position or by keyword. See [board-geometry.md](https://tabletwin.com/docs/board-geometry/#3-own-functions) |

## 5. Methods

Only these, and without keyword arguments:

| On | Method |
|---|---|
| dict | `.get(k)`, `.get(k, default)`, `.items()`, `.keys()`, `.values()` |
| list, tuple, text | `.count(x)`, `.index(x)` (`ValueError` if absent; on text `x` must be text) |

`.items()`, `.keys()` and `.values()` return lists. Any other method is an `AttributeError` at
run time if the name is one of the six, and a syntax error otherwise.

## 6. Names in scope

Every expression sees the **global part**:

| Name | Value |
|---|---|
| constants | Under their own name |
| variables, zones | Under their own name (a player zone is a list of lists) |
| derived values | Under their own name, computed when used |
| `players` | `[0, 1, …, n−1]` |
| `n_players` | Number of players |
| `eliminated` | List of eliminated players |
| `results` | `{subflow: result}` with result `'win'`, `'loss'` or `'draw'`, for scoped end conditions |
| `content` | The `content` layer |
| `modified` | See [§4](#4-built-in-functions) |
| board functions, own functions, builtins | See [§4](#4-built-in-functions) |

Then, depending on the place, **local names**:

| Place | Extra names |
|---|---|
| Flow: `over_x`, `until_x`, `when_x`, step `actor_x` and `visible_to_x` | None. Use the loop variable (`as`), which is a global variable |
| End conditions: `if_x`, `who_x`, `returns_x` | None |
| Action `pre_x` | `actor` (the step's actor, or who has priority in a window) |
| Ops in `cost` and `effect` | `actor`, plus every name bound by earlier ops (`bind` of `choose`, `roll`, `let`, `look`, `search`, `create`); names bound in `cost` stay for `effect` |
| `for_each` body | Also the `as` name (default `it`) |
| `search` `filter_x` | Also the item, as `it` (or `as`) |
| In a priority window | Also `window` (its name) and `stack` (list of `{action, actor, cancelled}`, bottom first); for a `window` op also the op's local names |
| `protocol.first_x` | For a window on a node: none. For a `window` op: the op's local names |
| Ability on a content item (`pre_x`, `cost`, `effect`) | As an action, plus `self` (the item) |
| `exec` effect | The local names of the `exec`, plus `self` (the item) |
| Trigger `if_x`, `controller_x`, `do` | The local names at the `emit`, plus `event`; `self` for a content trigger. On an `end_event` only `event` |
| Content `active_x` | `self`, plus the local names of the trigger or modifier being checked |
| Modifier `if_x`, `change_x` | `value` (the value so far), the keyword arguments of `modified()`, `self` for a content modifier, and for `modify` the local names at that time |
| Outcome `derived_x`, `p_x`, `effect` | The local names of the `sample`, plus earlier `derived_x` names |
| Agent `policy_x` | `options` (list of labels), `decider` (the player) |
| Variable `init_x` | Only: builtins, constants, `content`, `players`, `n_players`, `eliminated`, `results`, board and own functions, and variables declared above. No zones, derived values or `modified` |
| Zone `init_x` | As variable `init_x`, with all variables |
| Own function `let`, `x` | Its parameters and earlier `let`s, plus the global part; not the caller's local names |
| Tactic expressions | See [tactics.md](https://tabletwin.com/docs/tactics/) (`me`, `pool`, `chosen`, `o`, `i`, `opts`, `labels`, `where`) |

**Lookup order** when names collide: local names, then variables, then zones, then the global
names. Among the global names: `modified`, own functions, board functions, the reserved names
(`content`, `players`, `n_players`, `eliminated`, `results`), constants, builtins, in that order
of precedence. So a local `cell` hides the board function `cell()`, and a variable `max` hides
the builtin. Pick names that do not collide.

## 7. Twin fields `_x`

A field named `<name>_x` holds an expression; `<name>` without `_x` is text for people (or, for
some fields, a literal value). See [reference.md § Conventions](https://tabletwin.com/docs/reference/#1-conventions) for
which fields accept literals and what happens to text without `_x`.

All expression fields the engine reads:

| Where | Fields |
|---|---|
| Flow node | `over_x`, `until_x`, `actor_x`, `visible_to_x`; case `when_x` |
| Action | `pre_x` |
| Ops | `value_x`, `amount_x`, `of_x`, `what_x`, `at_x`, `n_x`, `with_x`, `die_x`, `options_x`, `min_x`, `max_x`, `actor_x`, `visible_to_x`, `filter_x`, `items_x`, `until_x`, `cond_x`, `target_x`, `change_x`, `if_x` (modify) |
| Trigger | `if_x`, `controller_x` |
| Modifier | `if_x`, `change_x` |
| Content item | `active_x` |
| Priority protocol | `first_x` |
| State | `init_x` (zone, var); `derived` (`x`, or the value itself) |
| Outcome model | `derived_x` (dict of expressions), `p_x` |
| Agent | `policy_x` |
| End condition | `if_x`, `who_x`, `returns_x` |
| Function | `let` (dict of expressions), `x` |

An `_x` value may be a YAML string, number or boolean. The validator compiles every `_x` field
with a text or number value, every `derived_x` and `functions` expression, and `derived` entries
written as `{x: ...}`, so their syntax errors show up before play. A derived value written as a
plain string is only compiled when it is first used.

Quote expressions in YAML when they contain `:`, `#`, `{`, `[`, or start with a quote:
`pre_x: "hand[actor] != []"`.

## 8. Errors

Syntax errors (reported by the validator):

| Message | Cause |
|---|---|
| `syntax error in expression "<src>": <what> (position <n>)` | Not valid syntax (`empty expression`, `unterminated string`, `comments are not allowed`, `f-strings are not allowed`, …) |
| `<Construct> is not allowed in "<src>"` | A forbidden construct: `Lambda`, `Yield`, `Await`, `assignment`, `NamedExpr`, `Starred`, `BitOr`, `BitAnd`, `BitXor`, `Invert`, `LShift`, `RShift`, `MatMult`, `Ellipsis`, `semicolon`, … |
| `attribute .<name> is not allowed in "<src>"` | An attribute other than the six methods |
| `name <name> is not allowed in "<src>"` | A name starting with `_` |

Run-time errors:

```
error evaluating '<src>': <Kind>: <message>
```

`<Kind>` is named as in Python: `TypeError`, `ValueError`, `KeyError`, `IndexError`,
`ZeroDivisionError`, `NameError` (unknown name at that place), `AttributeError`,
`RecursionError`. A few errors come without that wrapper: `dict keys must be text, not int`,
`string formatting with % is not allowed; use str() and +`.

## 9. Examples

```python
score[actor] >= GOAL
[c for c in hand[actor] if c['cost'] <= gold[actor]]
max(players, key=total_of)            # total_of from functions
sum(c['value'] for c in table)
{**unit, 'damage': 0}
'Ranged' in self.get('keywords', '')
sorted(hand[actor], key=card_value, reverse=True)[:2]   # card_value from functions
```

## 10. Differences from Python

The language follows Python's semantics, with these deliberate differences:

1. **One number type.** `2.0` and `2` are the same number: `str(4/2)` is `'2'` (Python
   `'2.0'`), and `L[4/2]` works (Python: `TypeError`). Use `int()` when an integer is meant.
2. **Dict keys are text.** `{1: 'a'}` and `dict([(1, 2)])` are errors. `k in d` with a non-text
   `k` is `False`.
3. **Key order.** Dicts keep insertion order, except text keys that are integers (`'2'`, `'10'`):
   they come first, ascending. Sort before relying on such an order.
4. **`range`, `.keys()`, `.values()` and `.items()` return lists.** `range(3) == [0, 1, 2]` is
   `True`.
5. **Set order** is insertion order (Python: arbitrary). Use `sorted()` when order matters.
6. **No `%` formatting on text.** Use `str()` and `+`.
7. **No complex numbers or bytes.** `(-8) ** 0.5` is `nan`.
8. **Integers above 2^53** lose precision.
9. **Dates in YAML** stay text.