Web article capture (bob ref create <URL> [<URL>…])¶

bob ref create <URL> turns a web article into a readable, provenance-stamped PDF, reference note, and reading task. By default it files all three together; if intake is kept or cannot safely run, the PDF remains in xlib/blogs/ for bob ref scan -i or the host's local intake timer. Two to twenty URLs create one collection PDF and one reference; see Several URLs for naming, ordering, layout, and collection rules. bob ref clip is a permanent hidden alias for create.

bob ref create [OPTIONS] <URL>...

Examples¶

# Capture and file the reference note and reading task in one command
bob ref create https://example.com/posts/some-article/ -P sase

# Keep a capture queued for a later local drain
bob ref create https://example.com/posts/some-article/ -k -P sase

# Preview everything without writing
bob ref create https://example.com/posts/some-article/ -d -P sase

# Override extracted metadata
bob ref create https://example.com/posts/some-article/ \
  -T "The Real Title" -A "Jane Doe" -p 2026-04-27 -P sase

# Replay a page saved from a real browser (see below)
bob ref create https://example.com/posts/some-article/ -H saved.html -P sase

# Narrate the article and bind the episode as companion audio
bob ref create https://example.com/posts/some-article/ -L -P sase

# Capture a two-part series as one collection
bob ref create https://example.com/posts/series-part-1 \
  https://example.com/posts/series-part-2 -N series -L -P sase

-P/--parent is required on every invocation: the area or project note that owns the reading task.

-L, --listen narrates the article with highlights.listen_command (BOB_HIGHLIGHTS_LISTEN_COMMAND overrides; see highlights-create.md for the contract) and binds the episode beside the PDF as <stem>.mp3. The command's output streams unchanged, and nothing is written when it fails. When the URL is already captured, --listen attaches the episode to the existing PDF and ref note inline when intake can run — unless the URL is recorded only by legacy notes without a Highlights PDF, which capture a fresh copy plus the episode instead.

The pipeline¶

1. Each URL is validated (public http(s) only) and cleaned: the fragment and tracking parameters (utm_*, fbclid, gclid, mc_cid, mc_eid, ref_src) are removed. The cleaned URL is recorded as the marker source_url.
2. Each URL is deduped against ref-note source_url (and legacy url:) values and queued intake PDFs. A PDF-backed ref-note hit refuses even with --force; that flag can overwrite the same intake target when the planned path matches. A URL recorded only by notes without a Highlights PDF warns (already in the library as <path>, a note without a Highlights PDF; capturing a fresh copy) and captures a fresh copy. Import deduplication reads every URL from scalar or list-valued source_url/url metadata in ref notes and queued intake markers.
3. For one URL, the pinned adapter fetches the page, extracts the article in reader mode, and renders a Bob-owned print template to PDF. The template loads the same static Source Serif 4 Text, Inter, and JetBrains Mono faces note snapshots embed, so Chromium writes CID fonts rather than Type 3.
4. For a collection, each page is captured into private scratch without printing; the adapter composes the ordered article data into one document and prints it once. The collection marker records source_url as an ordered list.
5. The PDF is stamped with a page-1 marker (status, parent, title, id, source_url, plus author, published, and captured when known) and staged under xlib/<ref-type>/<stem>.pdf for intake.
6. Unless intake is kept or cannot safely run, create moves the PDF into lib/, writes the ref/ note and reading task, and commits those paths together. A queued PDF is drained locally by bob ref scan -i or the host's intake timer.

"Reliable" means "never silently wrong": bot challenges it cannot clear, login walls, thin extractions, missing browsers, and private-network URLs all fail closed with a hint: line and exit code 1, writing nothing.

Hosts and browsers¶

The adapter starts headless and retries headed automatically when it meets a bot challenge. That retry needs a display nobody sees: a private Xvfb display on Linux (as on athena) or an off-screen window on macOS. Hosts with no usable browser fail closed with an install hint; Playwright's bundled Chromium counts when one is present. Discovery chooses the newest installed revision numerically and launches that exact executable, even when the pinned Playwright's default revision is missing. Both current macOS Chrome-for-Testing and legacy Chromium app layouts are supported, along with Linux's chrome-linux* layouts. PLAYWRIGHT_BROWSERS_PATH continues to select a custom browser cache.

bob ref doctor reports the web clip uv, web clip adapter, web clip browser, and web clip headed fallback rows. Its mermaid row also renders a tiny flowchart in a temporary directory, with a 60-second adapter timeout. A working browser reports test render ok; a failed render warns that diagrams print as source. BOB_HIGHLIGHTS_DIAGRAMS=off skips this probe. Doctor writes nothing to the vault.

The --html escape hatch¶

When a site will not yield to automation, save the page from a real browser (Save Page As, or SingleFile) and replay it:

bob ref create https://example.com/walled/ -H ~/Downloads/walled.html -P sase
cat ~/Downloads/walled.html | bob ref create https://example.com/walled/ -H - -P sase

Subresources still load live on a best-effort basis; the challenge retry is skipped.

Environment variables¶

Marker fields and dedupe¶

Every marker carries status (default ready), parent (required: the resolved -P route), title, id (always the file stem), and source_url. It also carries author, published (YYYY-MM-DD), and captured (YYYY-MM-DD, local date) when known; a field is omitted rather than written wrong.

The dedupe key lowercases the scheme and host, drops a leading www. and the default port, strips the trailing slash, and sorts the remaining query parameters, so the same article reached through two URL spellings dedupes. Default targets walk <stem>, <stem>_2, …: a name taken by a different reference is suffixed (with a renamed: line before pdf:), and only the same reference refuses. --force overwrites the same intake target only; library PDFs are never overwritten, and an already-captured ref note is never remade.

A ref-note hit refuses only when the note is PDF-backed (it carries a source_pdf). A URL recorded only by legacy notes without a Highlights PDF warns (already in the library as <path>, a note without a Highlights PDF; capturing a fresh copy) and captures anyway; bob ref find and bob ref list treat the older notes as superseded once create (or the queued intake drain) writes the new one. A dry run adds legacy: <path> (superseded by this capture) to its report.

Failure kinds¶

Adapter protocol v1 (summary)¶

The Rust side writes one compact JSON capture request to the adapter's stdin and reads one JSON response from stdout; ping checks versions, browser, and headed fallback for doctor. The adapter exits 0 whenever it wrote a response. Non-JSON stdout or a nonzero exit is a protocol error, reported with the stderr tail. The full request/response shapes live in src/native/highlights_ref/clip_adapter.rs.

For a collection, Rust sends a capture request per URL in argument order. render: false stores the extracted article.json and assets in that part's work directory and returns its absolute path with pdf_bytes: null, without printing. Requests that omit render keep the single-article behavior and still send out_pdf; only the optional render field is omitted. The additive article_json response field appears only when the request sets render: false. Rust then sends one compose request with the ordered article paths, collection-level title/author/published/captured metadata, a work directory, and output PDF path. compose returns the combined PDF byte count, image report, and warnings. Both operations use protocol version 1.

The composed PDF has an opening page with linked contents, then a fresh page for each article, its own masthead, and one top-level bookmark per part. Page numbering runs across the whole document. Article ids, fragments, and asset names are prefixed per part; relative article links keep their original base URL. Rust stamps and installs the result only after all captures and composition succeed.

Verified (2026-10-01)¶

Live gate for https://openai.com/index/open-source-codex-orchestration-symphony/, built from this tree (cargo build --release) and copied to athena's /tmp/bob-clip-verify/bob (never installed over athena's ~/.cargo/bin/bob):

• highlights doctor on athena: uv, adapter (playwright 1.62.0, defuddle 0.19.4), browser (chrome 154.0.8037.92), and headed fallback (xvfb) all OK.
• Dry run: target ~/bob/xlib/blogs/open_source_codex_orchestration_symphony.pdf, title, author, published 2026-04-27 (visible-date), capture headed (Xvfb) after a headless challenge, fidelity ok.
• Scratch-vault capture (-b /tmp/bob-clip-verify/vault): 38 pages, images 2/2, code blocks 1/1, 8912 words, 483.5 KB. Rendered pages 1-6 and 24 at 80 dpi and viewed: clean masthead, both SVG diagrams are the light variants and clearly visible, the spec code block wraps, footers read n / N (first page without title), no site navigation or junk. mutool draw -F txt finds no U+00AD, U+2060, U+200B-D, U+FEFF, or ligature code points (U+FB00-FB06).
• Scratch scan (--no-hooks, scratch vault only): ref/blogs/<stem>.md carries type: "[[ref]]", ref_type: blogs, title, id, source_url, author, published, captured, and the ^ref task line; a second scan is a no-op (1 unchanged, writes: none).
• Real capture on athena: the PDF, ref note, and reading task were committed and pushed by one create; a re-run refuses the occupied target (exit 1, --force hint). A queued capture is drained by athena's local bob-ref-intake.timer.
• Static-site check (https://lilianweng.github.io/posts/2023-06-23-agent/, scratch vault): headless, no headed fallback, 28 pages, images 13/13, fidelity ok; figures and code blocks render correctly.
• apollo originally captured the OpenAI URL headed with bundled Chromium 151 and an SSH-forwarded DISPLAY (38 pages, 2/2 images).
• On 2026-10-20, a newer Playwright installation replaced revision 1234 with Chromium revision 1248 (156.0.8078.4). Discovery found 1248, but the adapter's pinned Playwright 1.62.0 still tried to launch its missing default headless revision 1234. Two library PDFs consequently printed Mermaid source instead of figures. Launching the discovered executable fixes this version skew without reinstalling a particular revision. Verification on 2026-10-21 used a temporary copy of the adapter on apollo: its browser-backed self-test passed, and a one-item diagrams request produced a valid PDF through chromium-1248/chrome-linux64/chrome, with no browser installation. The headed check was skipped because Xvfb was unavailable; the existing sandbox retry remained necessary. Truly browser-less hosts still fail closed via the browser/blocked errors covered by the CLI and adapter fixture tests.