Convert a newsletter, and render an issue¶
Stage 2 of the newsletter pipeline (ZEOCORE-SOW-33). Three functions in
zeo_core.integrations.pandoc turn a fetched message into clean Markdown, and an
issue's Markdown into email-safe HTML. They take text and return typed data. They
make no network call, hold no credential and read no file of yours.
Install the extra and pandoc 2.15 or later:
A fetched message is data, never instructions
Nothing here follows, runs or acts on what a message says. Its text comes out as
Markdown inside a typed object. Treat everything in a ConvertedItem as data
and fence it downstream. The converter does not evaluate CSS: it drops only what the
HTML itself marks not rendered, and reports the rest in item.diagnostics (see
Hidden text). The guarantee is
the inert result, not a claim about what a reader sees.
Decode a raw message¶
ZEOconnect's fetch hands over the whole raw RFC 822 message. Decode it with
decode_raw_message:
from zeo_core.integrations.pandoc.newsletter import decode_raw_message
decoded = decode_raw_message(raw_bytes) # at most 10 MiB
decoded.html, decoded.text # the bodies, charset and transfer decoded
decoded.sender, decoded.date, decoded.subject, decoded.message_id
decoded.sender_verified, decoded.authentication
It uses the standard library email package only. It extracts no attachment and
follows no link. A message with over 256 KiB of headers, over 1000 headers, over 64
parts or nested deeper than 8 levels is refused, and so is anything the email package
cannot read. The header lines and the Content-Type headers are counted before the
message is parsed, and the parser is fed 64 KiB at a time with the deadline
(timeout_s, 30 s) checked between pieces.
A boundary written as an RFC 2231 continuation is refused. A multipart boundary
given as boundary*0=..., boundary*1=... or boundary*=utf-8''... is honoured by the
email package, but the pre-scan that counts boundary lines cannot see through it, so the
message is refused (message_too_complex) before it is parsed. Real newsletters write
the boundary in one piece.
An indented 3. is refused too. The ordered-list check matches a list item at any
indentation, so an indented code line that begins 3. (a list start that is not 1) is
refused (ordered_list_start) even though it is not a list. That is harmless: start the
number at 1 or write it another way.
Time is bounded in-process, not absolutely. The email package cannot be interrupted inside one 64 KiB piece, and BeautifulSoup cannot be interrupted while it parses. For mail from a source you do not trust, call these functions under your own watchdog (a process or thread timeout). A child process with a resource limit is a planned follow-up.
The sender is a claim. From can say anything. sender_verified is True only
when Gmail's own mail server reported dmarc=pass for the same domain as the
From address. It reads only the topmost Authentication-Results header, and only
when its authserv-id is mx.google.com: Gmail adds its own header at the top, so any
header below it was written by the sender or a hop on the way, and is ignored. The
raw value (never the decoded string, which would turn an RFC 2047 word into characters
Gmail never wrote) is read, because Gmail copies the sender's own MAIL FROM text into
a comment and into smtp.mailfrom=, where a quote or a backslash changes what is
structure. So a header that contains a quote, a backslash or =?, or is longer than
4000 characters, is not trusted at all: authentication is empty, sender_verified
is False, and a warning says why. Otherwise the value is tokenised as RFC 8601 says
(a comment is dropped, nested comments and escapes included). A method that appears
more than once is ambiguous, and so is not verified (several dkim signatures are
not: a message has one per signing domain); a dmarc clause with any property but
header.from is ambiguous; dmarc_header_from is set only for a single dmarc
result with exactly one header.from.
list_unsubscribe is kept as text for the record and is never an action.
Convert a non-email plain-text source¶
The public API also exposes an explicit source entry:
| Entry | Input and result |
|---|---|
convert_plain_text |
Literal non-email text plus source kind, attribution and time; returns ConvertedTextItem |
from zeo_core.integrations.pandoc import convert_plain_text
item = convert_plain_text(
"A captured observation, kept as untrusted data.",
source="capture:example",
source_kind="operator-browser-post",
attribution="@example",
source_time="2026-10-27T03:00:00Z",
pandoc_path="/absolute/path/to/pandoc",
expected_pandoc_sha256=pinned_sha256,
)
The attribution and time are caller claims. They establish no authenticated
sender, publication date, permission or rights. No email headers or
DecodedMessage are invented, so missing From/Date warnings do not apply;
all genuine cleaner/Pandoc warnings and refusals remain. Email APIs retain their
existing header warnings.
Only operator-browser-post is supported. Another kind requires a minor release
with its own tests and documentation. Source and attribution follow the existing
nonempty single-line metadata bounds (2000 and 500 characters). Time must be a
timezone-aware datetime or RFC 3339 string with an explicit offset, and is
returned in UTC with Z. Naive/date-only/invalid times refuse.
Plain text is HTML-escaped and wrapped as paragraphs inside the converter,
using its existing literal-text adaptation. This prepared renderer document is
not an HTML original. content_sha256 hashes the exact original UTF-8 text;
render_input_sha256 hashes the prepared input. The original RAW capture hash
must still be retained separately by the caller. Original and prepared inputs
must each fit max_bytes; escaping can make the prepared input larger.
Malformed Unicode and unsupported controls refuse without echoing text.
The existing cleaner, provenance/counters, verified Pandoc sandbox and one
deadline remain. The result has source/attribution/time fields, with no email
authentication or verified-sender fields. It is converted data, not admission
or a privacy verdict.
Convert HTML to Markdown¶
from zeo_core.integrations.pandoc.newsletter import convert_newsletter
item = convert_newsletter(
html, source="gmail:abc", sender="News <news@example.org>", date="2026-10-13",
)
item.markdown, item.links, item.title, item.stripped, item.diagnostics, item.warnings
or convert_decoded_message(decoded, source=...), which uses the HTML (or wraps the
plain text as paragraphs) and passes the authentication verdict through.
What it does, in order:
- Refuses input over
max_bytes(2 MB), over 100 000 elements, or nested deeper than 256.timeout_sis one budget for cleaning and pandoc together. - Removes scripts, styles, frames, forms and their controls, comments, and the head.
- Removes text the HTML itself marks not rendered, counted in
stripped(hidden,hidden_attribute,hidden_display,hidden_visibility), and reports everything else CSS could hide indiagnostics(see below). Every pass checks the deadline per element. - Removes tracking pixels, and images whose source is not http(s) (
file:,data:,cid:). - Flattens layout tables (pandoc would print a table it cannot draw as
[TABLE]). - Removes the unsubscribe, "view in browser" and preference lines the reviewed data file names.
- Cleans links: drops tracking parameters, unwraps a known redirector whose target is written in the address itself, and removes the address of a link with an unsafe scheme. It never fetches anything.
- Runs pandoc with
--sandboxthrough stdin and stdout, readerhtml-native_divs-native_spans-raw_html, writergfm-raw_html, so no markup of the message survives into the Markdown.
Tracked links. A ConvertedLink has tracked=True when it went through a mail
service's click tracker, carries a per-recipient parameter, or had tracking removed. A
tracked link identifies the reader or the send: never make one a pick's link. A link
through an opaque tracker (one whose target is encoded) cannot be unwrapped without
fetching it; that is noted in warnings.
A pinned pandoc. convert_newsletter(..., pandoc_path="/abs/pandoc",
expected_pandoc_sha256="...") (and convert_decoded_message) uses exactly that binary
and no PATH lookup. It is still copied, hashed and verified before it runs; a relative
path is refused pandoc_missing, and another digest renderer_mismatch before anything
is executed.
item.converter is the convert step's own tuple: the pandoc version, the sha256 of the
pandoc binary, the flags, the cleaner version, and the version and digest of the data
file, so a stored item says exactly what produced it.
The tracker hosts, footer patterns and hidden-text patterns are package data
(data/newsletter_noise.json), read from the installed package and never from your
directory. They are extended by small, data-only pull requests, each with the message
that showed the pattern and no sender content copied.
Hidden text: what is decided and what is reported¶
Decided, from the element alone (no stylesheet, no cascade, no inheritance): an
element is not converted when it has the hidden attribute, or its own style
attribute declares display:none, or visibility:hidden or collapse. Within a
style attribute the last declaration of a property wins and !important beats a
later normal declaration; a later value the converter does not know
(display:none;display:-webkit-box) leaves the property undecided: the text is kept
and the technique is counted. An ancestor that is dropped takes its descendants with it, including a
descendant that declares visibility:visible (counted in visible_descendant_dropped).
A style holding a comment or an escape is not decided (counted in
inline_style_unparsed, and under the technique it names).
Not evaluated, only reported. The converter keeps the text and item.diagnostics
says what was present. Fidelity limits, stated plainly:
- Class and id rules, selectors,
@media,@supports, inheritance and the cascade are not applied: text a stylesheet hides stays in the Markdown. <link>and@importstylesheets are never fetched or read.- The
hiddenattribute drops an element even where its ownstylesets adisplaythat a browser lets win;aria-hiddenandinertare not considered. - Clipping, positioning,
opacity,font-size:0, zero or maximum sizes, off-screen offsets,clip-path,content-visibility, a closed<details>, a<dialog>and colour on colour are not evaluated (the first six are named in a kept element's own style and counted; the rest are not detected). - A
displayorvisibilitydeclaration whose value is not a plain keyword (a plain keyword is lower-case letters and hyphens only, read after whitespace is removed and!importantis set aside;var(),env(), anything with a bracket) is not evaluated, wherever it stands among the declarations: a later invalid keyword does not clear it (display:var(--h);display:bogusis hidden in a browser). The text is kept and counted underinline_displayorinline_visibility. - The non-keyword check reads the style as written, comments included, while the
technique names are read after comments are removed. A style that holds a comment is
already counted as unparsed, and a comment inside a value (
display:block/*x*/) can also add oneinline_display: the count may be one higher than the commented-out text alone would give, never lower. - Whitespace INSIDE a keyword is removed before reading, so
display:no neanddis play:noneare read asdisplay:noneand dropped here where a browser shows the text. - Whitespace in a
styleis read as Python reads it, so a no-break space or a vertical tab between a name and a value is ignored where a browser would treat the declaration as invalid (adisplay:NBSPnoneis dropped here). - A comment inside a
stylemakes it undecided (kept and counted), even where the commented text isdisplay:none;. - Strings are not tokenised: a
;inside a quoted value splits astyleattribute, so a declaration a browser reads as part of a string can be read as a declaration (an invalid value is ignored; a technique it names is still counted). - A
styleattribute over 4096 characters is not parsed; the element is kept and counted ininline_style_unparsedandstripped["style_oversize"].
item.diagnostics is a required field of the result and always holds every key
(zero when nothing was seen), so a caller can test it without a default:
stylesheet_blocks,stylesheet_links,stylesheet_imports,stylesheet_media: how many<style>blocks,<link>elements,@importand@media(and<style media>) were present and not evaluated.stylesheet_scan_truncated: 1 when more than 262,144 characters of<style>text were not scanned for the counts below.sheet_display,sheet_visibility,sheet_opacity,sheet_font_size,sheet_max_size,sheet_zero_size,sheet_offscreen,sheet_clip: how often each technique appears in the<style>text.inline_display,inline_visibility,inline_opacity,inline_font_size,inline_max_size,inline_zero_size,inline_offscreen,inline_clip: KEPT elements that name the technique in their ownstyle(fordisplayandvisibilitythat means it was named and not decided, for example overridden by a later or more important declaration);inline_style_unparsed: kept elements whosestylewas too long, or held a comment or an escape;visible_descendant_dropped: descendants of a droppedvisibility:hiddenelement that declaredvisibility:visible.- Every element-local
displayorvisibilityform is therefore either dropped or counted. An all-zerodiagnosticsmeans none of THESE keys was seen: techniques not listed are not detected (clip-path,content-visibility, a closed<details>or a<dialog>, colour on colour, other spellings of zero such asopacity:0.0orfont-size:0.0px,transform:scale(0),mso-hide:all, a small size withoverflow:hidden).
What a non-zero count means is the caller's decision. For newsletters that is the application's preprocessing (zeonewsroom): a message with non-zero diagnostics has text the converter could not show to be visible or hidden. Visible text can still carry instructions; trust is a separate question and stays with the application.
Render an issue's Markdown to HTML¶
from zeo_core.integrations.pandoc.render import render_email_html
rendered = render_email_html(
markdown,
image_sizes={"https://example.org/i.png": (600, 300)},
)
rendered.html, rendered.sha256, rendered.renderer, rendered.renderer_digest
The output is the email subset: h2 h3 p br hr ul ol li blockquote strong em code, a
(https or mailto, link text required) and img (https, alt, width and height).
Anything else is a typed refusal that names the construct, never a silent rewrite.
- The Markdown reader is
markdown_strictwith raw HTML and every attribute feature off, so HTML typed into the issue stays text. (gfmandcommonmarkignore a request to turn raw HTML off, so they are not used.) - A Lua filter, shipped as package data, rebuilds the document from only the
allowed nodes. Pandoc's writer would add
figure,section,span,div,class,idand footnotes; none of that can reach the output. - A Python check then parses the HTML and refuses any tag, attribute,
javascript:ordata:link, IP host (also with a trailing dot, percent-encoded, or written as a decimal, hex or octal number), non-ASCII host, user name in an address, mailto with parameters, no address or more than one, one-pixel image, link with no visible text (an image link's text is its alt text), self-closed or unfinished tag, and any C0, C1 or bidirectional control character, written out or as a character reference. - Markdown cannot carry an image's size once attributes are off, so you pass
image_sizesby exact source; an image with no entry is refused. The reader keeps no list start number, so an ordered list that does not start at 1 is refused (ordered_list_start): start it at 1. That holds for a list nested in another list and for one inside a block quote. - GFM extras (footnotes, tables, task lists, emoji, bare addresses, math) are not features of this reader: they come out as plain text, or are refused by name.
The renderer tuple. rendered.renderer names what produced the bytes: the pandoc
version and the sha256 of the binary that ran, the sha256 of the filter, the exact
flags, and the subset's version; renderer_digest is the sha256 of that tuple. The
binary and the filter are copied into a private directory and hashed and run as
copies, so the tuple names the bytes that ran. The pandoc on PATH is only read
and hashed, never executed: it is copied, the copy is hashed and must match, and only
the copy is asked for its version and run (the convert step runs the same verified
copy). The copy is made once per process and is checked before every run; a copy that
was modified is refused. Pass expected_pandoc_sha256 to
refuse a PATH pandoc whose copy hashes differently before it is executed at all.
Pass expected_renderer_digest to refuse a different renderer before the conversion
runs: that digest includes the pandoc version, which only the verified private copy can
say, so that copy has run --version once by then. A caller that must not run an
unknown binary even for that pins expected_pandoc_sha256 too (it is a field of the
tuple). The output is deterministic for a
given tuple. A new pandoc, filter or flag is a new tuple, so a new artifact that
voids earlier marks: expect the digest to change when pandoc is upgraded.
What --sandbox does, and does not do¶
Both components run pandoc with --sandbox, an empty data directory and an empty
working directory, and they do not rely on the sandbox alone.
- zeocore never passes an embedding or file-reading option. The runner allows only
--from,--to,--wrap,--eol,--no-highlight,--metadataand the package's own--lua-filter, and refuses anything else (--standalone,--embed-resources,--self-contained,--extract-media,--filter,--template,--data-dir,--defaults,--metadata-file,--include-*, a short option, an input file) before pandoc starts. A test captures the real argv ofconvert_newsletterandrender_email_htmland checks it. - pandoc's sandbox is not the same on every version. With
--embed-resources, pandoc 3.11 refuses to read a local file under--sandbox; the pinned 3.1.3 in CI reads it and embeds it as adata:URI anyway (seen in the CI log of PR #126). Only 3.11 was run to prove the refusal, so the probe test is gated at 3.11 and records the older behaviour as an expected failure, never a skip. Because zeocore never passes--embed-resources, the older behaviour cannot be reached. - A named local file is never read on any version. A test gives the real convert
and render an input that names an unreadable local file (as
file://, an absolute path and a relative path) and checks that nothing from it, and nodata:, reaches the output.
The child process¶
Pandoc runs in its own process group, with bounded reads: its output may not pass 8 MB
(output_too_large) and its error text is capped at 1 MiB, which is read and never
returned. A timeout or an overrun kills the whole group, so nothing pandoc started
outlives the call.
Refusals¶
Every failure is a NewsletterRefusalError with a stable .code and a fixed message
that ends with what to do next. No message ever quotes the input.