# Web article capture (`bob ref create […]`) `bob ref create ` turns a web article into a readable, provenance-stamped PDF, reference note, and reading task. By default it files all three together; if intake is kept or cannot safely run, the PDF remains in `xlib/blogs/` for `bob ref scan -i` or the host's local intake timer. Two to twenty URLs create one collection PDF and one reference; see [Several URLs]() for naming, ordering, layout, and collection rules. `bob ref clip` is a permanent hidden alias for `create`. ``` bob ref create [OPTIONS] ... ``` ## Examples ``` # Capture and file the reference note and reading task in one command bob ref create https://example.com/posts/some-article/ -P sase # Keep a capture queued for a later local drain bob ref create https://example.com/posts/some-article/ -k -P sase # Preview everything without writing bob ref create https://example.com/posts/some-article/ -d -P sase # Override extracted metadata bob ref create https://example.com/posts/some-article/ \ -T "The Real Title" -A "Jane Doe" -p 2026-04-27 -P sase # Replay a page saved from a real browser (see below) bob ref create https://example.com/posts/some-article/ -H saved.html -P sase # Narrate the article and bind the episode as companion audio bob ref create https://example.com/posts/some-article/ -L -P sase # Capture a two-part series as one collection bob ref create https://example.com/posts/series-part-1 \ https://example.com/posts/series-part-2 -N series -L -P sase ``` `-P/--parent` is required on every invocation: the area or project note that owns the reading task. `-L, --listen` narrates the article with `highlights.listen_command` (`BOB_HIGHLIGHTS_LISTEN_COMMAND` overrides; see [`highlights-create.md`]() for the contract) and binds the episode beside the PDF as `.mp3`. The command's output streams unchanged, and nothing is written when it fails. When the URL is already captured, `--listen` attaches the episode to the existing PDF and ref note inline when intake can run — unless the URL is recorded only by legacy notes without a Highlights PDF, which capture a fresh copy plus the episode instead. ## The pipeline 1. Each URL is validated (public `http(s)` only) and cleaned: the fragment and tracking parameters (`utm_*`, `fbclid`, `gclid`, `mc_cid`, `mc_eid`, `ref_src`) are removed. The cleaned URL is recorded as the marker `source_url`. 2. Each URL is deduped against ref-note `source_url` (and legacy `url:`) values and queued intake PDFs. A PDF-backed ref-note hit refuses even with `--force`; that flag can overwrite the same intake target when the planned path matches. A URL recorded only by notes without a Highlights PDF warns (`already in the library as , a note without a Highlights PDF; capturing a fresh copy`) and captures a fresh copy. Import deduplication reads every URL from scalar or list-valued `source_url`/`url` metadata in ref notes and queued intake markers. 3. For one URL, the pinned adapter fetches the page, extracts the article in reader mode, and renders a Bob-owned print template to PDF. The template loads the same static Source Serif 4 Text, Inter, and JetBrains Mono faces note snapshots embed, so Chromium writes CID fonts rather than Type 3. 4. For a collection, each page is captured into private scratch without printing; the adapter composes the ordered article data into one document and prints it once. The collection marker records `source_url` as an ordered list. 5. The PDF is stamped with a page-1 marker (`status`, `parent`, `title`, `id`, `source_url`, plus `author`, `published`, and `captured` when known) and staged under `xlib//.pdf` for intake. 6. Unless intake is kept or cannot safely run, `create` moves the PDF into `lib/`, writes the `ref/` note and reading task, and commits those paths together. A queued PDF is drained locally by `bob ref scan -i` or the host's intake timer. "Reliable" means "never silently wrong": bot challenges it cannot clear, login walls, thin extractions, missing browsers, and private-network URLs all fail closed with a `hint:` line and exit code 1, writing nothing. ## Hosts and browsers The adapter starts headless and retries headed automatically when it meets a bot challenge. That retry needs a display nobody sees: a private Xvfb display on Linux (as on athena) or an off-screen window on macOS. Hosts with no usable browser fail closed with an install hint; Playwright's bundled Chromium counts when one is present. Discovery chooses the newest installed revision numerically and launches that exact executable, even when the pinned Playwright's default revision is missing. Both current macOS Chrome-for-Testing and legacy Chromium app layouts are supported, along with Linux's `chrome-linux*` layouts. `PLAYWRIGHT_BROWSERS_PATH` continues to select a custom browser cache. `bob ref doctor` reports the `web clip uv`, `web clip adapter`, `web clip browser`, and `web clip headed fallback` rows. Its `mermaid` row also renders a tiny flowchart in a temporary directory, with a 60-second adapter timeout. A working browser reports `test render ok`; a failed render warns that diagrams print as source. `BOB_HIGHLIGHTS_DIAGRAMS=off` skips this probe. Doctor writes nothing to the vault. ## The `--html` escape hatch When a site will not yield to automation, save the page from a real browser (Save Page As, or SingleFile) and replay it: ``` bob ref create https://example.com/walled/ -H ~/Downloads/walled.html -P sase cat ~/Downloads/walled.html | bob ref create https://example.com/walled/ -H - -P sase ``` Subresources still load live on a best-effort basis; the challenge retry is skipped. ## Environment variables | Variable | Meaning | | --- | --- | | `BOB_WEB_CLIP_ADAPTER` | Replaces the adapter invocation (test seam) | | `BOB_CHROME` | Browser executable the adapter launches instead of discovery. Discovery itself tries Google Chrome at the usual Linux and macOS paths, then Playwright's bundled Chromium, newest installed revision first, launched by its discovered path even if Playwright's default revision is missing | | `BOB_WEB_CLIP_TIMEOUT_SECS` | Overall adapter timeout in seconds (default 300) | | `BOB_WEB_CLIP_KEEP_WORKDIR` | `1` keeps the scratch directory for debugging and prints its path | ## Marker fields and dedupe Every marker carries `status` (default `ready`), `parent` (required: the resolved `-P` route), `title`, `id` (always the file stem), and `source_url`. It also carries `author`, `published` (`YYYY-MM-DD`), and `captured` (`YYYY-MM-DD`, local date) when known; a field is omitted rather than written wrong. The dedupe key lowercases the scheme and host, drops a leading `www.` and the default port, strips the trailing slash, and sorts the remaining query parameters, so the same article reached through two URL spellings dedupes. Default targets walk ``, `_2`, …: a name taken by a different reference is suffixed (with a `renamed:` line before `pdf:`), and only the same reference refuses. `--force` overwrites the same intake target only; library PDFs are never overwritten, and an already-captured ref note is never remade. A ref-note hit refuses only when the note is PDF-backed (it carries a `source_pdf`). A URL recorded only by legacy notes without a Highlights PDF warns (`already in the library as , a note without a Highlights PDF; capturing a fresh copy`) and captures anyway; `bob ref find` and `bob ref list` treat the older notes as superseded once `create` (or the queued intake drain) writes the new one. A dry run adds `legacy: (superseded by this capture)` to its report. ## Failure kinds | Kind | Meaning and fix | | --- | --- | | `blocked` | Challenge or login wall; rerun headed, or pass `--html FILE` | | `thin` | Extraction too small; retry with `--html`, or not an article | | `browser` | No usable browser. The hint is `install Google Chrome or set BOB_CHROME=/path/to/chrome`. A system Chromium package needs `BOB_CHROME` | | `network` | DNS, TLS, HTTP, or private-network refusal | | `timeout` | Capture took too long; raise `BOB_WEB_CLIP_TIMEOUT_SECS` | | `unsupported_content` | Non-HTML, non-PDF content | | `render` | Printing failed | | `invalid_request` | Malformed adapter request (a bug; report it) | ## Adapter protocol v1 (summary) The Rust side writes one compact JSON `capture` request to the adapter's stdin and reads one JSON response from stdout; `ping` checks versions, browser, and headed fallback for `doctor`. The adapter exits 0 whenever it wrote a response. Non-JSON stdout or a nonzero exit is a protocol error, reported with the stderr tail. The full request/response shapes live in `src/native/highlights_ref/clip_adapter.rs`. For a collection, Rust sends a `capture` request per URL in argument order. `render: false` stores the extracted `article.json` and assets in that part's work directory and returns its absolute path with `pdf_bytes: null`, without printing. Requests that omit `render` keep the single-article behavior and still send `out_pdf`; only the optional `render` field is omitted. The additive `article_json` response field appears only when the request sets `render: false`. Rust then sends one `compose` request with the ordered article paths, collection-level title/author/published/captured metadata, a work directory, and output PDF path. `compose` returns the combined PDF byte count, image report, and warnings. Both operations use protocol version 1. The composed PDF has an opening page with linked contents, then a fresh page for each article, its own masthead, and one top-level bookmark per part. Page numbering runs across the whole document. Article ids, fragments, and asset names are prefixed per part; relative article links keep their original base URL. Rust stamps and installs the result only after all captures and composition succeed. ## Verified (2026-10-01) Live gate for `https://openai.com/index/open-source-codex-orchestration-symphony/`, built from this tree (`cargo build --release`) and copied to athena's `/tmp/bob-clip-verify/bob` (never installed over athena's `~/.cargo/bin/bob`): - `highlights doctor` on athena: uv, adapter (playwright 1.62.0, defuddle 0.19.4), browser (chrome 154.0.8037.92), and headed fallback (xvfb) all OK. - Dry run: target `~/bob/xlib/blogs/open_source_codex_orchestration_symphony.pdf`, title, author, published 2026-04-27 (visible-date), capture headed (Xvfb) after a headless challenge, fidelity ok. - Scratch-vault capture (`-b /tmp/bob-clip-verify/vault`): 38 pages, images 2/2, code blocks 1/1, 8912 words, 483.5 KB. Rendered pages 1-6 and 24 at 80 dpi and viewed: clean masthead, both SVG diagrams are the light variants and clearly visible, the spec code block wraps, footers read `n / N` (first page without title), no site navigation or junk. `mutool draw -F txt` finds no U+00AD, U+2060, U+200B-D, U+FEFF, or ligature code points (U+FB00-FB06). - Scratch scan (`--no-hooks`, scratch vault only): `ref/blogs/.md` carries `type: "[[ref]]"`, `ref_type: blogs`, `title`, `id`, `source_url`, `author`, `published`, `captured`, and the `^ref` task line; a second scan is a no-op (1 unchanged, writes: none). - Real capture on athena: the PDF, ref note, and reading task were committed and pushed by one create; a re-run refuses the occupied target (exit 1, `--force` hint). A queued capture is drained by athena's local `bob-ref-intake.timer`. - Static-site check (`https://lilianweng.github.io/posts/2023-06-23-agent/`, scratch vault): headless, no headed fallback, 28 pages, images 13/13, fidelity ok; figures and code blocks render correctly. - apollo originally captured the OpenAI URL headed with bundled Chromium 151 and an SSH-forwarded `DISPLAY` (38 pages, 2/2 images). - On 2026-10-20, a newer Playwright installation replaced revision 1234 with Chromium revision 1248 (156.0.8078.4). Discovery found 1248, but the adapter's pinned Playwright 1.62.0 still tried to launch its missing default headless revision 1234. Two library PDFs consequently printed Mermaid source instead of figures. Launching the discovered executable fixes this version skew without reinstalling a particular revision. Verification on 2026-10-21 used a temporary copy of the adapter on apollo: its browser-backed self-test passed, and a one-item diagrams request produced a valid PDF through `chromium-1248/chrome-linux64/chrome`, with no browser installation. The headed check was skipped because Xvfb was unavailable; the existing sandbox retry remained necessary. Truly browser-less hosts still fail closed via the `browser`/`blocked` errors covered by the CLI and adapter fixture tests.