Reading the organization lenses

Which lens for which question

Run a CLI lens over the files of one program. The MCP tool api_organize accepts page, stack, conventions, aid, layers, effects, or writes, returning the same text; layers also returns package and definition edges from typed references, with direct effect sites and tables per package; effects and writes also return the rows their tree is drawn from. The implementations lens returns the structured nested program map and implementation comparison. In WebMCP, pass the source array directly:

api_organize({
  sources: [{fileName: "forum.bosatsu", source: fileContents}],
  lens: "conventions"
})

Use api_abstraction_map for the visual dependency map in the WebMCP workbench. Its browser view can group definitions by package and checked type, then focus on direct users and dependencies. Try it on ProjectHub. The groups use checked signatures, not inferred responsibilities; sharing a type does not prove that definitions implement the same policy.

The layers effect list locates primitive operations in each package’s own definitions. A package with no direct sites may still call another package’s IO code.

Author a reading; check it against the program.

An author can choose which blocks to bring forward and add explanatory captions. The JSON view format below works with the text lenses: the generated page stays intact, and a checked authored view appears below it. It does not control the browser diagram’s layout.

This view brings the shared stock rule into the foreground alongside its callers. The names come from the generated stack for the repaired stock program.

{
  "lens": "stack",
  "order": ["cart_units", "checkout_units", "support_units"],
  "foreground": ["fulfillable_units"]
}

Download stock.view.json. From a built checkout, run:

java -jar target/scala-3.8.2/yichus.jar organize view \
  --view web/evidence/diagram-examples/stock.view.json \
  demos/organization/stock-after.bosatsu

The checker resolves each name and requires every root of this stack to remain in the authored view. Change the foreground name to missing_rule and rerun: the view is refused because the program has no such definition. The generated page remains available.

For declared groups and reading orders, the tools also compare membership and dependency order with the compiled program. These are specific structural checks. Free-form captions stay under says:; their formatting and references are checked, but their meaning is not proved. A complete authored browser diagram with arbitrary proved prose claims remains a design direction.

The Forum browser uses a separate implementation-view-1 document. It declares nested groups, pins implementations, and names the reference targets for each comparison dimension. The engine resolves qualified identities, rejects duplicate ownership and cyclic containment, and regenerates the complete user-reference graph. Group labels and role meanings remain authored descriptions.

Its automatic layout orders peers to reduce crossings and horizontal travel when the search finds an improvement. Optional order entries such as group:representation or node:Yichus/Examples/Forum::respond override peer order. Dependency bands remain; placement does not claim conceptual abstraction levels. Edit view.json in the example and rebuild to check an override.

When several packages define the same name, labels include the shortest package suffix that distinguishes them. For example, ContributionModel::jobs_table and JudgmentStore::jobs_table remain separate rows. Selection and metadata always use full qualified identities. A frozen comparison chooses labels across both snapshots, so switching sides does not rename a kept definition.

Follow an indirect dependency. Choose Show paths in a comparison cell. Each hop of a shortest reference path opens its exact source witnesses. These are static user references, which can include passing functions as values. They do not prove execution or permission enforcement. The inspector distinguishes an absent path from an incomplete search. Select a definition and choose Focus direct neighbors to see its immediate users and references, with its authored ancestors and a whole-program locator.

Review a refactor. Choose Freeze before snapshot, edit the source or choose a candidate, then rebuild. The comparison retains the question and pins. Definition, qualified-type, visibility and reference changes appear separately from authored grouping and ordering changes. Matching source excerpts establish only equal text; they do not establish equivalent behavior. The editor and Run button use the current version.

For a moved definition, edit correspondence.json with its exact old and new identities. Each endpoint can occur once, and both must exist. Without a mapping, changed identities appear as additions and removals. A mapping is the author's correspondence, not an equivalence proof:

[{
  "before": "Icetakes/LawyerPreparation::same_contribution",
  "after": "Icetakes/ContributionModel::same_contribution"
}]

Share what you inspected. Download selected evidence exports the question, pins, full identities, static relation kinds, source witnesses, snapshot identity and comparison context. A recorded example also links its full artifact. A freshly edited snapshot has no hosted artifact link; save its source and checked artifact together if someone else must reproduce it. WebMCP exposes this same bundle as diagramView.selectedEvidence.

The inspector's Ownership, declarations and direct effects disclosure distinguishes the defining package from an authored group. It also shows the written targets of FamilySpec and ReadingOrder declarations beside their resolved identities. The existing resolver tries an exact qualified identity, then a bare name in the declaration's package, then a bare name unique across the program. Missing or ambiguous names fail. A resolved target identifies the definition; it does not prove the declaration's prose description.

To embed these views in another application, use the program-map consumer contract. Agents working on the example should read its instructions with yichus_example_read, edit against the returned revision, rebuild, and inspect yichus_example_feedback. The page's yichus_page_feedback and yichus_page_action tools expose the same selection, freeze and snapshot controls available to a person.

The vocabulary page: concepts and the roles bindings play

The one question this page answers: what conceptual vocabulary is this program written in, and what role does each binding play in it.

A concept is a declared type of the program (a struct, an enum, a newtype). Under each concept the page sorts the bindings that mention it by role: intro makes one (the concept appears only in the result), build transforms one (both sides), use consumes one (only in the arguments), const is a value of it. ~internal marks a binding its package does not export. A capability is a struct of operations; the forum declares none, and the page says so.

java -jar target/scala-3.8.2/yichus.jar organize vocabulary demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-vocabulary.txt. Lines elided are marked ....

$ yichus organize vocabulary demos/service/forum.bosatsu
type-vocabulary  forum.bosatsu
concepts 12  capabilities 0  bindings 60  plumbing 24 (no user type in signature; framework artifacts counted separately)  framework artifacts 5  bare-scalar params 41 (Int 7, String 34)
...
concepts (declared types, by bindings served):
   Post  struct  bare-scalar params: Int x0, String x3
      intro: posts_table  (db: Yichus/Data::Db) -> Yichus/Data::Table[List[Post]]
      build: replace_post  (posts: List[Post], edited: Post) -> List[Post]  ~internal
      build: find_post  (posts: List[Post], target: String) -> Option[Post]  ~internal
...

What it does not say. What a concept means. It knows Post is a struct that seven bindings serve; it does not know what a post is.

The stack page: blocks by layer, in reading order

The one question this page answers: in what order does a reader learn this program's building blocks, and how much of the program does each block explain once learned.

Every top-level definition is a block. A block's layer is one above the highest block it is built from, so layer 0 blocks are built from nothing in the program and the last layer holds the entry points; read bottom layer first, and each layer is the layers below it pieced together. Each itemized line gives the block's name, its checked signature, and:

• used-above N: how many higher blocks build from it (its fan-in). A block at used-above 0 is a root: an entry point used by nothing above.
• plumbing: no declared concept in its signature (learned by its body, not its type); ~internal: not exported by its package.
• [framework]: a declaration the platform consumes (an access rule set, a declared backing system); [flow]: an instance of a declared mutation flow.
• roots: the line under the header names every root, itemized or not, so the header's count, a layer's remainder, and an itemized used-above 0 line are never three places to look for the same fact.
• [test]: a root whose type is Test, a test entry rather than the program's; the header counts them apart (roots 21 (...; tests 8)) so a reader at scale can tell the program's entry points from its tests' without reading package names.
• built from: every block below it that its body calls, complete. A platform external (a table read or write, a permission) is not a block and never appears, so a block that reads a table shows the table accessor it calls, not the read.
• concepts first spoken here: declared types whose lowest serving block is in this layer; layer skips: edges that reach more than one layer down.
• A layer itemizes at most twelve blocks, by fan-in; the rest are counted, never dropped silently: (+9 more blocks in this layer, not itemized). A root that falls in the unitemized remainder is still counted in the header; the remainder line says how many blocks the cap hides.

java -jar target/scala-3.8.2/yichus.jar organize stack demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-stack.txt. Lines elided are marked ....

$ yichus organize stack demos/service/forum.bosatsu
stack  forum.bosatsu
layers 8  blocks 60  roots 6 (used by nothing above: the entry points)  concepts 12  layer skips 96
roots: access_rules, forum_api, forum_reading, forum_shape, forum_systems, forum_tables
...
L0  24 blocks  plumbing 10  used-above total 47  concepts first spoken here: Issue, Outcome, Post, Profile, Thread, Visibility
   moderators_table  (db: Yichus/Data::Db) -> Yichus/Data::Table[List[String]]  used-above 5  plumbing ~internal
   posts_table  (db: Yichus/Data::Db) -> Yichus/Data::Table[List[Post]]  used-above 3
   (+12 more blocks in this layer, not itemized)
...
L1  11 blocks  plumbing 6  used-above total 50
   jraw  (key: String, body: String) -> String  used-above 11  plumbing ~internal
   has_room  (thread: Thread, replies: List[Post]) -> Bool  used-above 1  ~internal
...
L4  6 blocks  plumbing 1  used-above total 6  concepts first spoken here: EditPostRequest, OpenThreadRequest, PutProfileRequest, ReadThreadRequest, ReplyRequest
   post_reply  (db: Yichus/Data::Db, p: Yichus/Access::Principal, req: ReplyRequest) -> Yichus/IO::IO[Yichus/Service::Response]  used-above 1
L7  1 blocks  plumbing 1  used-above total 0
   forum_api  Yichus/Service::ServiceDef  used-above 0  plumbing

The stack drawn

organize stack --svg --focus can_see draws a definition and the definitions it depends on, transitively. Full names stay visible. Arrows point toward dependencies; a wire that skips a layer uses an outer lane. Hover a wire for its exact endpoints. The legend counts the omitted blocks and edges, and empty rows above the focused region are cropped without changing layer spacing. An unknown or ambiguous name is refused.

The rows, families and dependencies come from the program. Ordering, spacing and routing are choices made by the renderer. Crossings and visual density therefore do not establish that a program is well organized. Use the source behind an edge, the conventions page, and a before/after comparison to judge a proposed change.

An earlier cold-reading experiment compared images of whole-program version pairs. It did not establish that the drawing let readers distinguish their organization reliably. The original experiment and failed results remain unchanged. Focused dependency inspection and standalone export are narrower jobs; a readable export is not evidence of whole-program comprehension.

java -jar target/scala-3.8.2/yichus.jar organize stack --svg --focus can_see \
  demos/service/forum.bosatsu --output can-see.svg

Verbatim from docs/forum-demo/outputs/step1d-organize-stack-focus.svg. Download the focused SVG.

A paired reading pilot asked independent agents to assess dependency, layer, and scope statements about this Forum focus. One reader received the complete user-package source; the other read the exported SVG as text, with identical questions and public lens rules. Both answered all seven statements correctly. Including the common sheet, the SVG packet was 6,929 UTF-8 bytes and the source packet was 25,012. These are input sizes, not measured model tokens. This selected example does not test rendered-image comprehension, behavioral refactoring, or whole-program organization.

Method and limitations; questions; frozen protocol and material hashes; recorded verdict; SVG reader answers; source reader answers.

A separate fresh pair assessed the same structural statements and materials through the read-only Codex CLI, with the packets supplied inline and no tools used. Both answered all seven correctly. The runtime reported 17,426 input plus output tokens for the SVG reader and 21,338 for the source reader, including the CLI's input context. Producing and maintaining the export remain outside this comparison.

Frozen runtime-token protocol; runtime trial verdict; SVG reader receipt; source reader receipt; SVG reader event stream; source reader event stream.

Still to evaluate: cold reading of rendered focused images across more programs. The earlier whole-program image failure remains the evidence for that broader task.

java -jar target/scala-3.8.2/yichus.jar organize stack --svg \
  demos/service/forum.bosatsu --output forum-stack.svg

The page ends with a reading order section. A program may declare, as a typed binding, the order a reader should learn the blocks. The declarations the stack, conventions, and flow pages check are values of the builtin package Yichus/Organize, written in the program itself; a family, a reading order, and an expectation look like this, with each string naming a binding or a resource of the program and each caption authored, not measured (except a count of routes, tables, handlers, or steps a caption states, which the page checks):

from Yichus/Organize import FamilySpec, ReadingOrder, ReadingStep, Expectation, AtomicRmw

handlers = FamilySpec("handlers", ["open_thread", "post_reply", "edit_post"])
how_to_read = ReadingOrder([
  ReadingStep("jraw", "the one JSON primitive every renderer uses"),
  ReadingStep("respond", "how a handler turns a view into a response"),
])
posts_rmw = Expectation("posts", AtomicRmw)

The page checks the order against the layers and prints the result. A populated section prints one line per declared step with one of three outcomes: convergent (the step comes after every block it is built from), divergent (it is listed before a block it is built from, and those blocks are named), or absent (no such binding); the section also names any root the order leaves out. The forum declares a reading order of thirty-three steps (forum_reading), and the page checks every step against the layers, printing each step's measured layer and built-from blocks under the declaration's own caption:

Verbatim from docs/forum-demo/outputs/step1c-organize-stack.txt. Lines elided are marked ....

reading order (declared, Yichus/Organize::ReadingOrder; checked against the layers above):
...
   forum_reading  steps 33  convergent 33  divergent 0  absent 0  roots not listed 0  blocks not listed 22
      1. quote  L0
         built from (nothing on this page)
         says: wraps a string in JSON quotes; every JSON line below uses it
      2. join_with  L0
         built from (nothing on this page)
         says: joins strings with a separator; the JSON arrays and objects are built on it
...

The says: line under each step is the declaration's caption, authored; the layer and the built-from blocks beside it are the page's own measurement, which a caption cannot change. One thing in a caption is checked: a count of routes, tables, handlers, or steps it states (eight routes, 2 tables) is compared with the program's, and a mismatch prints under the line as count: the caption says eight routes; the program's ServiceDef declares 9 routes. A caption whose counts hold, or that states none, prints nothing more.

What it does not say. What a block does: the page shows signatures, layers, and fan-in, never a body. To learn what has_room decides you open has_room.

The conventions page: how each recurring shape is written

The one question this page answers: how does this project write each recurring shape, and where does it disagree with itself.

A family is two or more bindings of one signature shape whose type fillers agree on every slot but at most one type, or whose holes are positions that vary together (lockstep holes: two slots vary in lockstep when every pair of members agrees on one slot exactly when it agrees on the other, so (CommentWire) -> Outcome[Comment] and (TaskWire) -> Outcome[Task] form one family with two holes). Each binding joins the largest family it fits; identical signatures cluster first, one-type holes second, lockstep holes only among the bindings the first two left alone. The family line gives the shape with the fixed fillers named and holes as _, the member count, and how many members take each form. Under it, one line per member, grouped by form so a member written another way sits apart:

• form: the compiled body's root after lambdas and lets: literal-match (a match on a literal), variant-match (a match on a constructor), computed-branch (a test that is neither), call, construct, loop, factory (returns a function), value (a constant or an accessor).
• builds from: the program's own definitions the body calls.
• reads: the fields of the member's parameter concepts its body destructures or projects, in declaration order. A member with no reads line either reads nothing of its concepts or reads through a newtype, whose one field compiles away and is not measured; the page cannot tell those apart, so treat an absent reads line as "not stated".

A second table lists each concept with the families it has a member in, so a concept lacking a shape its neighbours have is visible by comparing rows. The page ends with declared families: a program may declare, as a typed binding, that some bindings are written the same way, and the page checks the claim against the families it inferred, printing per member whether its shape converges with the family, diverges (the differing shape named), or the name is absent, and stating the split of forms. The forum declares two families, handlers and tables, and the page refuses each claim in part: two of the eight declared handlers take no request and so have another shape, and two of the four declared tables hold lists.

java -jar target/scala-3.8.2/yichus.jar organize conventions demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-conventions.txt. Lines elided are marked ....

$ yichus organize conventions demos/service/forum.bosatsu
conventions  forum.bosatsu
families 18 (2+ bindings of one signature shape agreeing on every filler but one type, or but fillers that vary in lockstep)  concepts served 11
...
how this project writes each shape:
   (Db, Principal, _) -> IO[Response]  x6  forms: call 3, variant-match 3  concepts: EditPostRequest, GrantModeratorRequest, OpenThreadRequest, PutProfileRequest, ReadThreadRequest, ReplyRequest
      grant_moderator  call  builds from forbid, is_moderator, jarr_of_str, jraw, member_String, moderators_table, respond, roster_key
      open_thread  call  builds from jarr, jraw, parse_thread, reject, render_thread, respond, threads_table
      read_thread  call  builds from can_see, forbid, jarr, jint, jraw, moderators_table, not_found, posts_table, render_post, render_thread, respond, roster_key, threads_table
      edit_post  variant-match  builds from can_edit, can_see, check_text, find_post, forbid, jarr, jint, jraw, moderators_table, not_found, posts_table, reject, render_post, render_thread, replace_post, respond, roster_key, threads_table  reads thread_id, post_id, body
      post_reply  variant-match  builds from can_see, check_text, forbid, has_room, jarr, jint, jraw, moderators_table, not_found, posts_table, reject, render_post, render_thread, respond, roster_key, threads_table  reads thread_id, body
      put_profile  variant-match  builds from check_text, jarr, jraw, profiles_table, reject, render_profile, respond
...
   (Db, Principal) -> IO[Response]  x2  forms: call 2
      get_profile  call  builds from jarr, jraw, profiles_table, render_profile, respond
      list_threads  call  builds from can_see, jarr, jint, jraw, moderators_table, render_thread, respond, roster_key, threads_table
...
   (List[Post], _) -> _[Post]  x2 (lockstep holes)  forms: call 1, loop 1  concepts: Post
      replace_post  call  builds from post_id_is  reads id
      find_post  loop  builds from post_id_is
...
concepts and the families they have a member in:
   Post  (_) -> String; (Db) -> Table[List[_]]; (List[Post], _) -> _[Post]; (String, List[String], _) -> Bool; (_, List[_]) -> _
...
declared families (Yichus/Organize::FamilySpec, checked against the families above):
...
   handlers  declared 8  convergent 6  divergent 2  absent 0  not claimed 0
      family (Db, Principal, _) -> IO[Response]  forms: call 3, variant-match 3 (no single form: the declared members disagree among themselves)
      convergent  put_profile, grant_moderator, open_thread, read_thread, post_reply, edit_post
      divergent   get_profile  shape (Db, Principal) -> IO[Response]; the family's is (Db, Principal, _) -> IO[Response]
      divergent   list_threads  shape (Db, Principal) -> IO[Response]; the family's is (Db, Principal, _) -> IO[Response]
   tables  declared 4  convergent 2  divergent 2  absent 0  not claimed 0
      family (Db) -> Table[_]  forms: call 2
      convergent  profiles_table, threads_table
      divergent   posts_table  shape (Db) -> Table[List[_]]; the family's is (Db) -> Table[_]
      divergent   moderators_table  shape (Db) -> Table[List[_]]; the family's is (Db) -> Table[_]

What it does not say. Which form is right. Six handlers split three call and three variant-match; the page shows the split and marks nothing.

The aid: a labeled heuristic over the conventions page

The one question this page answers: I cannot judge from the conventions page; which member should I compare with its neighbour.

The aid is a separate program over the conventions page for a reader who cannot judge from it. Every line it prints carries the label heuristic:, and its header says the rest: a flag is a place to compare, never evidence. It flags a member whose reads lack a field that a same-family neighbour reads; it states how many members it could not compare and why.

java -jar target/scala-3.8.2/yichus.jar organize aid demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-aid.txt. Lines elided are marked ....

$ yichus organize aid demos/service/forum.bosatsu
conventions-aid  forum.bosatsu
HEURISTIC AID: a program over the conventions page for a reader who cannot judge from the page.
Every line below is a heuristic, never evidence. A flag names a member whose reads set lacks a
...
families read 18  members read 46 (35 not compared: no reads line, or an accessor)  flags 0
heuristic: (none: no compared member lacks a field its own concept declares and a same-family neighbour reads)

What it does not say. That an unflagged program is right: flags 0 with 35 members not compared is a statement about what the aid could see.

The abstraction map: dependencies and definition sizes

The one question this page answers: what is built from what, by abstraction height, and where the duplicated mass sits.

Each line is a definition with its layer, s (compiled size), in (times used), out (direct dependencies), and, when present, dup m/g g/i i (duplicated mass inside the definition, in g repeated shapes over i occurrences). Below a definition, its branches: span is how many layers the edge reaches down, and a span over one is marked SKIP. [+N below] counts a collapsed subtree. (framework: discovered by type) and (exported) mark declarations and API surface so in0 out0 never reads as dead code.

Branches are drawn with +- for a branch that has siblings below it and \- for the last one. A zoomed page opens with a zoom: line counting the definitions shown: expanded ones show their branches, ones at the frontier are shown with their branches collapsed, and vocabulary counts declared types the zoom pulled in. Zoom with --depth N for an overview of the roots' first N levels, --focus NAME for one definition, and --inline NAME to show an abstraction's branches at each call site.

java -jar target/scala-3.8.2/yichus.jar organize map demos/service/forum.bosatsu --depth 1

Verbatim from docs/forum-demo/outputs/step1c-organize-map.txt. Lines elided are marked ....

$ yichus organize map --depth 1 demos/service/forum.bosatsu
abstraction-map  forum.bosatsu
layers 8  defs 60  edges 150  skips 96  max-size 217  dup 37  clone-pairs 0
legend: s=compiled size, in=times used, out=direct deps; a branch's span is how many layers it reaches (1 clean, >1 SKIP)
legend: dup<m>/<g>g/<i>i = m duplicated mass inside the def, g repeated shapes, i occurrences; [+N below] = N defs in the collapsed subtree (zoom in with focus <name>)
zoom: overview depth 1 — showing 14 of 60 defs (6 expanded, 8 at the frontier, 0 vocabulary)
...
L7  forum_api        s184 in0 out8  dup22/2g/4i  (framework: contains declared API routes)
   +- get_profile  span 4  SKIP (reaches 4 layers)
   \- open_thread  span 1
L6  open_thread      s70 in1 out7  (exported)  [+28 below]
L4  post_reply       s207 in1 out16  (exported)  [+36 below]
L0  access_rules     s41 in0 out0  (framework: discovered by type)

What it does not say. Which duplicated mass is worth removing. dup22/2g/4i on forum_api is eight route lines sharing two shapes; the map measures it, and the reader decides.

The flow page: effects in causal order, under declared systems

The one question this page answers: what happens when it runs, which resource each step touches, and which declared guarantee covers it.

One flow per binding that performs effects; the flow header names the binding and, in brackets, the package it lives in ([Demo/Forum]). Each step is an effect with the resource it touches in brackets: read, write, create, delete, mint id (a fresh identifier from a table, counted as neither a read nor a row write), init (a state cell brought into being), random (a value drawn from the platform's random source), batch (several effects submitted as one), or capture (a value recorded for the run's evidence); | between adjacent steps is a recorded causal order, and . no recorded order marks adjacent steps the graph does not order. << RMW on X >> flags a read causally before a write on the same resource; the legend says atomicity is not implied.

The header states the declared systems: a program may declare, as a typed binding, which backing system serves each resource (a Postgres database at a given isolation, an S3 bucket, a queue). The declaration is the program's own claim about where it will run; nothing on the page checks it against a live database, and the headline is what the named system promises, not what was observed. With a declaration, every step that touches a resource gets a ~ sys: line carrying only that system's derived headline; a resource no declaration covers prints UNDECLARED -- no guarantees; a queue-declared resource hit by a database step prints MISMATCH. Without any declaration the header says systems: none declared -- guarantees unknown, hazards NOT ruled out. The header's count, touched resources covered C of T, is taken over the flows on the page: T is the distinct resources any effect step touches, and C is those whose declared system governs database-style steps, so an undeclared resource and a queue-declared resource hit by a database step both count as touched and not covered. Under a read-modify-write, a further ~ rmw: line says the transaction boundary is not shown and atomicity is not attested, whatever the headline says. The expectations section is where a program's declared expectations of a resource would be checked; the forum declares none.

java -jar target/scala-3.8.2/yichus.jar organize flow demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-flow.txt. Lines elided are marked ....

$ yichus organize flow demos/service/forum.bosatsu
flow  forum.bosatsu
legend: | = recorded causal order between adjacent steps; '. no recorded order' =
adjacent steps the graph does not order (branch arms, sequence/batch members);
r=read w=write; a resource in [..] is the cell/table touched
legend: << RMW >> flags a read-then-write on the same resource; atomicity is NOT implied
systems: 4 declared; touched resources covered 4 of 4; ~ sys: lines carry ONLY declared guarantees

...
flow: post_reply   [Demo/Forum]
  read  [moderators]
    ~ sys: postgres forum: serializable -- txns serialize; RMW safe within one txn
    |
  read  [threads]
    ~ sys: postgres forum: serializable -- txns serialize; RMW safe within one txn
    |
  read  [posts]
    ~ sys: postgres forum: serializable -- txns serialize; RMW safe within one txn
    |
  mint id  [posts]
    ~ sys: postgres forum: serializable -- txns serialize; RMW safe within one txn
    |
  write  [posts]   << RMW on posts >>
    ~ sys: postgres forum: serializable -- txns serialize; RMW safe within one txn
    ~ rmw: read and write are separate operations; txn boundaries not shown -- atomicity NOT attested here
  (end)

What it does not say. That the write in post_reply is safe. The headline names what serializable Postgres gives a transaction; the ~ rmw: line says this lens cannot see the transaction.

The effects page: each route as a tree down to its effects

The one question this page answers: what each declared route can do, and what surrounds each effect it reaches: the calls that lead to it, the branches it sits in, the transaction around it, and the definitions it is handed to as an argument.

Each route names its handler, then a tree in source order. A node runs whenever the node above it runs, unless it is a branch: Bosatsu has no early return, so siblings with no branch above them all run. if X and unless X are branches, where X is a Bool call with its arguments or value is Constructor, read from the typed IR. transaction holds what runs inside one database transaction. (argument) holds an IO value handed to the call above it, which runs where that callee's subtree says runs <parameter>. The writes page is the same tree with only row writes drawn, and --summary adds counts as an index into the tree.

java -jar target/scala-3.8.2/yichus.jar organize effects demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-effects.txt. Lines elided are marked ....

$ yichus organize effects demos/service/forum.bosatsu
effects  forum.bosatsu
...
/moderators/grant -> grant_moderator
   |-- read [moderators]  :443
   |-- if roster is EmptyList  :444
   |   \-- write [moderators]  :446
   \-- unless roster is EmptyList  :444
       \-- if is_moderator(roster, uid)  :449
           \-- write [moderators]  :453
/posts/edit -> edit_post
   \-- unless check_text("body", body, 500) is NonEmptyList  :521
       |-- read [moderators]  :524
       |-- read [threads]  :525
       \-- unless row is None  :526
           \-- if can_see(uid, roster, thread)  :529
               |-- read [posts]  :530
               \-- unless find_post(posts, post_id) is None  :531
                   \-- if can_edit(uid, roster, post)  :534
                       \-- write [posts]  :536
...

What it does not say. Whether a write should sit under a guard or a wrapper. edit_post's write is under can_see and can_edit; that it is the right check for this data is a declared property the access checker proves, not something this page reads. The same facts are available as rows: api_organize returns them under effects, and an explore --query reads them as route_effects(data).

The organization diff: what changed between two snapshots

The one question this page answers: what changed, in the facts the other pages measure, between a before and an after.

Given the files of a before snapshot and an after snapshot, the page reports the movement of every headline channel (bindings, plumbing, dup mass, total mass, clone pairs, bypasses) and itemizes: bindings added and removed with their signatures; retyped (same name, changed signature); visibility (a kept binding whose export status flipped); concepts added, removed, or reclassified; capabilities; family joins and leaves (membership change in a family of the conventions page, the one family notion the diff and the page share, so a renderer added among (_) -> String members joins that family under the page's label); bypasses (the defs behind the headline's count, each with the bindings that hand-roll it, as the organization page lists them; a kept bypass whose hand-rollers changed prints with what they were); stack movement (a kept binding whose layer or used-above on the stack page changed: moved up 1 (L1 -> L2), used-above 1 -> 2, one line per binding and no aggregate, so collapsing a chain into a shared abstraction shows as the blocks that moved); framework artifacts, with the routes of a ServiceDef added, removed, or kept with a changed handler or permissions (~ route, with what it was); and declarations (a change to a claim about the program, kept apart from a change to the code). Identity is by name: the same name on both sides is the same binding. Under each heading, + opens an item that is new on the after side and - one that is gone from it.

The example diffs the forum's routes draft, the state before its request validation was written, against the finished forum. The draft is not a committed file: scripts/forum-demo/replay.sh rebuilds it from the patches under docs/forum-demo/steps/ and passes its path where the command below shows <routes draft>. To diff a change of your own, copy the forum file, edit the copy, and pass the two paths; either option may be repeated for programs of several files.

java -jar target/scala-3.8.2/yichus.jar organize diff --before <routes draft> --after demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-diff.txt. Lines elided are marked ....

$ yichus organize diff --before <routes draft> --after demos/service/forum.bosatsu
organization-diff  routes draft -> forum.bosatsu
bindings 56 -> 60 (+4)  plumbing 24 -> 24 (0)  dup mass 47 -> 37 (-10)  total mass 1994 -> 2261 (+267)
clone pairs 0 -> 0 (0)  bypasses 0 -> 0 (0)
legend: facts only -- judgment stays with the reader. The headline counts are
the organization page's: bindings = the program's top-level definitions
(framework artifacts included); plumbing = bindings whose signature touches no
declared concept; dup mass = compiled nodes repeated inside definitions instead
of shared, and total mass = the compiled nodes of every definition; clone pairs =
two definitions whose whole bodies are isomorphic, literals kept; a bypass = a
reused def hand-rolled elsewhere instead of called (the bypasses channel names
the def and the bindings that hand-roll it). A family join/leave
is membership change in a family of the conventions page (two or more
bindings of one shape agreeing on every filler but one type, or on fillers
that vary in lockstep; formed = the join
that made a family of two, joined = a join to a family of three or more); retyped =
same binding name, different checked signature; ~internal = not exported
by its package; visibility = a kept binding's export status flipped; stack
movement = a kept binding whose layer or used-above (the stack page's) changed
bindings added:
   + forum_reading  Yichus/Organize::ReadingOrder
   + forum_shape  Yichus/Organize::FamilySpec
   + forum_tables  Yichus/Organize::FamilySpec
   + thread_request_issues  (req: OpenThreadRequest) -> List[Issue]
bindings removed:
   (none)
retyped (same name, changed signature):
   (none)
visibility (kept bindings whose export status changed):
   (none)
concepts:
   (no change)
capabilities:
   (no change)
family joins:
   + forum_reading joined no family (a framework declaration: the conventions page lists it under framework artifacts, in no family)
   + forum_shape joined no family (a framework declaration: the conventions page lists it under framework artifacts, in no family)
   + forum_tables joined no family (a framework declaration: the conventions page lists it under framework artifacts, in no family)
   + thread_request_issues joined no family (2 other bindings share its shape (_) -> _[_], but it agrees with none of them on every filler but one: not a family)
family leaves:
   (none)
bypasses (a reused def hand-rolled elsewhere instead of called):
   (none)
stack movement (kept bindings whose layer or used-above changed):
   ~ forum_api  moved up 1 (L6 -> L7)  used-above 0 (unchanged)
   ~ open_thread  moved up 1 (L5 -> L6)  used-above 1 (unchanged)
   ~ parse_thread  moved up 1 (L4 -> L5)  used-above 1 (unchanged)
framework artifacts:
   + forum_reading: Yichus/Organize::ReadingOrder
   + forum_shape: Yichus/Organize::FamilySpec
   + forum_tables: Yichus/Organize::FamilySpec
   ~ route /posts/reply -> post_reply  read moderators, create posts, read posts, write posts, read threads  (was: post_reply  read moderators, read posts, write posts, read threads)
   ~ route /threads -> list_threads  read moderators, query threads  (was: list_threads  read moderators, read threads)
   ~ route /threads/open -> open_thread  create threads, write threads  (was: open_thread  write threads)
declarations (Yichus/Organize: a change to a claim, apart from a change to the code):
   + forum_shape  FamilySpec 'handlers' (8 members)
   + forum_tables  FamilySpec 'tables' (4 members)
   + forum_reading  ReadingOrder (33 steps)

The three ~ route lines say what the routes draft's permissions lacked: between the draft and the finished forum, replying gained create posts, listing threads moved from read threads to query threads, and opening a thread gained create threads. A route's handler and permissions are read from the compiled service definition, so a pass that adds a route, or widens one, shows here whatever its scope said. The forum's three declarations were written after the routes draft, so they appear three times: as bindings added (they are bindings), as framework artifacts (the platform consumes them), and under declarations, the channel that keeps a change to a claim apart from a change to the code and names what each claims. An added binding that joined no family says why, in the conventions page's own terms: a framework declaration, alone in its shape, or sharing its shape with bindings it agrees with on no every-filler-but-one set. The legend defines each count of the headline in the organization page's words, so bypasses 0 -> 0 (0) needs no other page to read, and the bypasses channel would name the def and its hand-rollers if the count moved.

What it does not say. Why the two tables left their family, or whether the leave is good. A body-only edit moves no membership fact; a shape change does, and the page names which.

The organization page: the glance block and the reuse sections

The one question this page answers: what are this program's measured properties? Counts describe reuse, duplication, and layering. To understand its organization, read the named blocks and their relationships in the stack, conventions, and structure views.

The page opens with a glance block: one line per channel, a count or a share, and a bar of # marks. Each # stands for one unit of a count (up to forty marks) or for five points of a share, rounded down; the marks are drawn from the digits, and the output's own line digits are authoritative says which to trust when they seem to differ. The channels are labeled cost (higher is worse: plumbing, dup mass, stamped mass, clone pairs, bypasses) or coverage (higher is better: concept-serving bindings, the bindings whose signature mentions a declared concept). Nothing combines them into a score.

• plumbing: bindings whose signature mentions no declared type of the program. Framework artifacts (declarations the platform consumes) are counted separately, never as plumbing.
• dup mass: compiled nodes repeated inside definitions instead of shared; clone pairs: two definitions whose whole bodies are the same with literals kept; bypasses: a definition that exists and is reused, hand-rolled again somewhere instead of called.
• stamped mass: a signature family is stamped when two or more of its members are copies of one compiled body with only the types changed, as if one skeleton had been stamped out per type; stamped mass is the compiled mass of those copied members, and the reuse line counts stamped signature families N of M over all families. An unmarked family shares a shape without sharing a body.
• concepts: one line per declared type with the count of bindings that introduce, build, use, or hold a constant of it (the roles the vocabulary page defines), their summed compiled mass, the duplication inside those bindings as dup N, and its bare-scalar parameters (raw Int or String positions a newtype would have named).
• signature families: bindings of one signature shape with different types filled in. A family line reads shape, then xN for N members, then which of the shape's type slots differ among the members: 2 of 3 slots vary, or identical signatures when none do. framework artifacts: bindings discovered by their checked type and consumed by the platform, listed so they never read as dead code.
• units and placeholders: every size or mass on any page (mass, dup, s) is a count of nodes in the compiled body; a _ in a shape is a hole, a position whose type differs among the members; a capability (counted beside the concepts) is a struct whose fields are operations, recognized by its field shapes, never by its name.

java -jar target/scala-3.8.2/yichus.jar organize page demos/service/forum.bosatsu

Verbatim from docs/forum-demo/outputs/step1c-organize-page.txt. Lines elided are marked ....

$ yichus organize page demos/service/forum.bosatsu
organization  forum.bosatsu
glance (cost, higher = worse):
   plumbing                   40%  ########
   dup mass                    1%
   stamped mass                0%
   clone pairs                  0
   bypasses                     0
glance (coverage, higher = better):
...
marks: each # = 1 for counts (up to 40); each # = 5 points of share, floor; digits are authoritative
...
reuse: dup mass 37 of 2261 total (1%)  clone pairs 0  bypasses 0  stamped signature families 0 of 6
plumbing: 24 of 60 bindings touch no declared concept (40%); framework artifacts (5) counted separately
vocabulary: 12 concepts (0 capabilities); 31 of 60 bindings serve one (51%); framework artifacts 5; bare-scalar params 41 (Int 7, String 34)
...
concepts (one line each, by bindings served):
   Issue  struct  intro 6  build 1  use 3  const 0  mass 208 dup 0  bare-scalars 13
   Post  struct  intro 1  build 2  use 4  const 0  mass 172 dup 0  bare-scalars 3
...
signature families (same shape, different types):
   (_[_]) -> _  x4  2 of 3 slots vary  mass 71 dup 0
...
framework artifacts (declarations discovered by checked type; not dead):
   access_rules: Yichus/Access::AccessSpec
   forum_systems: Yichus/Systems::SystemSpec
bypasses (a reused def hand-rolled elsewhere instead of called):
   (none)
clone pairs (whole bodies isomorphic, literals kept):
   (none)

What it does not say. Whether 40 percent plumbing is too much for a service of this size. The number is the fact; the reading is yours.

Build the CLI and run a lens

From a clone of the repository, with a JDK 17 or newer and sbt installed:

git clone https://github.com/snoble/yichus
cd yichus
sbt assembly
java -jar target/scala-3.8.2/yichus.jar organize stack demos/service/forum.bosatsu

Any lens takes one or more Bosatsu files that compile together as one program; a file that declares the same package name as one of the platform's builtin packages replaces it, and the first line of the output says so. To regenerate every example on this page: scripts/forum-demo/replay.sh (needs the jar and node).

Troubleshooting by outcome

• Error: program failed to compile: the files do not compile together; the message names the file and line. Fix the source there and run the command again; a lens never renders over a program that does not compile.
• no jar at target/scala-3.8.2/yichus.jar: run sbt assembly from the repository root.
• A section reads (none) or (none declared): the program has nothing of that kind, or declares nothing; both are stated absences, not errors.