Media Primitives

Status: Phase 10 plus v0.1.0 Alpha Wave 2 media implementation complete with deterministic and fake-server acceptance passing; live Telegram upload/download remains separately gated.

Scope

miniproto owns protocol-core media movement: chunked upload, messages.sendMedia, upload.getFile, CDN redirects, CDN reupload tokens, AES-CTR CDN chunk decryption, resumable destinations, and progress callbacks. Rich media convenience wrappers, bound message helpers, album helpers, thumbnail generation, and framework ergonomics remain future mpgram responsibilities.

Upload

Use Client.send_file(peer, file, caption="...") for the thin high-level path. It uploads the file with 512 KiB default parts, chooses upload.saveFilePart for small files and upload.saveBigFilePart for files over 10 MiB, then sends messages.sendMedia with inputMediaUploadedDocument by default. Optional quick_ack=True or quick_ack_callback=callback applies only to this final message-send RPC; it does not request a transport acknowledgement for every upload part and does not replace the final Telegram result. High-level uploads use dedicated MTProto sender lanes by default: the measured default is two upload lanes, so a socket reset on one media lane does not poison unrelated requests on the main client sender or every other in-flight part without opening one socket per concurrent part. Pass media_lanes=0 to force the legacy single-sender path, or pass an explicit positive media_lanes count to decouple socket count from part concurrency. Low-level callers can use miniproto.media.upload_file() directly to get the generated InputFile or InputFileBig object.

Uploaded or received document-like media and photos expose media.file_id, a miniproto-owned opaque ID that can be decoded later with decode_file_id() / media_from_file_id(). Passing that ID back to Client.send_file(peer, file_id, caption="...") sends inputMediaDocument or inputMediaPhoto and skips reuploading the bytes. Telegram represents videos, GIFs, audio, voice notes, stickers, and ordinary files as document media with different DocumentAttribute values, so miniproto file IDs use a document family for all of those and a separate photo family for photos. upload_file() enforces Telegram’s upload-part ceiling and can grow a caller-supplied small part_size up to the 512 KiB maximum to stay under max_file_parts; Client.send_file(..., upload_limit_parts="app_config") fetches the current help.getAppConfig value when a caller wants the live server ceiling instead of a fixed override.

Supported upload inputs are paths, bytes-like objects, seekable binary file objects, sync iterables of byte chunks, and async iterables of byte chunks. Unknown-size sources are spooled to a temporary file before upload because Telegram’s big-file method requires the total part count.

Download

Use Client.download_media(media, destination=None) for the thin high-level path. It resolves public Media, miniproto file IDs, raw MessageMediaDocument, raw MessageMediaPhoto, raw Document, raw Photo, or generated InputFileLocation objects into upload.getFile requests. Passing no destination returns downloaded bytes in MediaDownloadResult.data; passing a path writes to disk; resume=True appends to an existing path and starts requests after the existing byte count (partial tails past the last 1 KiB boundary are dropped and re-downloaded because Telegram cannot serve sub-1 KiB offsets). Downloads default to concurrency=6 pipelined requests across media_lanes=2 warm sender lanes with 512 KiB starting parts, adaptive part sizing up to the 1 MiB ceiling, and an 8 MiB rolling in-flight byte window — the bandwidth-delay product needed to saturate Telegram’s ~16 MiB/s per-transfer ceiling (mtcute ships 2 connections x 3 in-flight; MTKruto 2 x 2 x 1 MiB). media_lanes=0 reproduces the old main-sender path, and max_in_flight_bytes (or the legacy max_buffer_size alias) caps memory separately from request count.

The 8 MiB rolling window described above is per transfer. All simultaneous transfers created by one client additionally share byte-weighted schedulers keyed by (target DC, direction). Downloads and uploads have separate budgets, so full-duplex traffic does not consume the other direction’s capacity. Requests are charged in 64 KiB units around each live RPC attempt; retry and flood sleeps retain the transfer’s local fixed slot but release the shared byte permit, allowing unrelated transfers to progress without undoing the per-transfer pressure reduction. Deficit round-robin keeps simultaneous foreground transfers moving, read-ahead uses background priority, and background work receives a bounded anti-starvation grant. The defaults are 16 MiB of download requests and 8 MiB of upload requests per DC, configurable with ClientConfig.media_download_max_in_flight_bytes_per_dc and ClientConfig.media_upload_max_in_flight_bytes_per_dc; these are independent from media_max_buffer_size and max_in_flight_bytes.

For downloads, the scheduler also respects Telegram’s small/large same-DC file-operation guidance at the 20 MiB boundary. Explicit ClientConfig.media_download_small_queue_limit and media_download_large_queue_limit values take precedence; otherwise miniproto uses conservative 5-small/2-large fallbacks and updates them from small_queue_max_active_operations_count and large_queue_max_active_operations_count whenever the existing help.getAppConfig lifecycle supplies those values. A missing dynamic value never becomes zero capacity. FILE_MIGRATE_X moves the transfer registration and its future permits to the target DC without changing the main session.

Client.iter_download(media, ...) exposes the same ordered engine as an async byte iterator. It accepts the meaningful range, precision, part-size, concurrency, lane, retry, cache, read-ahead, reference-refresh, and integrity controls from download_media(), but it has no destination and does not support resume or bot multi_session; resume a stream by passing the desired offset. The first verified part is yielded before later requests complete, concurrent out-of-order results are held until their predecessors are available, and the active byte window bounds both live requests and the reorder buffer. Closing the iterator early with await iterator.aclose(), breaking an async for, or cancelling the consumer cancels owned part tasks and releases sender requests and scheduler permits. download_file() and download_media() consume this same engine into their threaded destination writer, so range legality, migration, refresh, CDN handling, retries, ordering, and integrity do not have a second implementation.

Bot downloads can opt into independent authorization-key sharding with multi_session=True. The measured policy uses one session through 50 MiB, two sessions through 250 MiB, and four sessions above 250 MiB. The option is disabled by default and applies only to complete, known-size bot downloads; user accounts, resumed/range downloads, and unknown-size inputs log an error and continue through the normal single-session path. Auxiliary sessions are created lazily only when a size threshold needs them, use one fixed 1 MiB request at a time, never poll or receive updates, disconnect after the transfer, and retain their session storage for later downloads and process restarts. With EncryptedSQLiteSessionStorage("bot.sqlite", ...), their stable names are bot-download-1.sqlite through bot-download-3.sqlite. Pass the bot token as ClientConfig(..., bot_token=token) when missing auxiliary sessions may need authorization after a restart; a token passed to sign_in_bot(token) is also retained in memory for the current client lifetime but is never persisted. Built-in in-memory and encrypted SQLite storage support auxiliary sessions; custom storage without sibling support logs an error and falls back to one session.

Media-lane pools are keyed by operation kind and target DC: asking for a different lane count resizes the existing pool instead of building a disjoint socket set, lanes are prewarmed in parallel at transfer start, and lanes idle for ClientConfig.media_idle_close seconds (default 120, None disables) are closed and transparently reopened on the next transfer. Media that lives on another DC is served by a dedicated pool for that DC using an exported authorization (auth.exportAuthorization -> per-DC auth key -> auth.importAuthorization on the media connection); FILE_MIGRATE_X re-resolves the pool without ever migrating the main session.

Every emitted upload.getFile request satisfies Telegram’s alignment rules: without precise, offsets and limits are 4 KiB-aligned, limits divide 1 MiB, and requests never straddle a 1 MiB boundary; miniproto switches individual transfers to precise mode automatically when 1 KiB granularity is needed (odd resume offsets or sub-4 KiB part sizes). part_size/max_part_size must be powers of two.

Concurrent downloads treat FLOOD_WAIT as per-request pacing, not congestion: the flooded request sleeps in place while HOLDING its window slot, so the request rate drops naturally while every other slot keeps flowing (mtcute’s fixed-slot model — backfilling freed slots with new requests sustains the rate the server just objected to and escalates the waits). Flood retries never consume the transient retry budget (a separate generous cap of 16 per part bounds storms), FLOOD_WAIT_0 sleeps for at least 1 s, and all flood sleeps add only up to 0.3 s of jitter so simultaneously-flooded requests do not retry in lockstep without adding hundreds of seconds of avoidable stall over a large transfer. The concurrency limit never shrinks on generic floods; only disconnect/timeout errors reduce by one, and FloodPremiumWait switches the current adaptive transfer to a single active request for the rest of that transfer because live user downloads showed it as the dominant throughput throttle signal. Burst-starts are damped by a bounded launch pacer: the first window uses a 50 ms stagger decaying by x0.8 to a 3 ms floor, after generic floods the pacer spaces new launches under the recent successful request rate without going below 4 launches/s, and pacing is skipped whenever the effective adaptive window is already one. launch_stagger=True is the production default; disabling it is an advanced benchmark control for paired matrix measurements, not a recommended throughput setting. Large downloads also use adaptive part sizing by default: if part_size starts below max_part_size, miniproto samples throughput, tries larger chunks up to the Telegram 1 MiB ceiling, and settles back to the best observed size if a larger chunk regresses. Set adaptive_part_size=False to keep the chunk size fixed, adaptive_concurrency=False to pin the window.

For range-style workflows, pass range_cache=True or a DownloadRangeCache instance plus an optional read_ahead_bytes budget. download_media() automatically uses a miniproto file ID as the range-cache key when one is available; raw download_file() callers can pass range_cache_key explicitly. Read-ahead is a streaming feature: full-file downloads force-disable it (with a warning metric) because prefetches racing the transfer’s own scheduler measured 3x slower, background prefetches are capped at 32 tasks, and disk writes ride a threaded writer pipeline on both the sequential and concurrent paths. Known-size concurrent downloads require every requested byte range to be received and committed before success; an empty or short response before the target end raises MediaDownloadError, removes a path created by that call, and restores a resumed path to its previously verified 1 KiB-aligned prefix. Progress callbacks are coalesced to at most one per 250 ms or 8 parts, with the final callback delivered only after complete coverage and the destination flush. The cache is in-memory and intentionally opt-in so full 2 GiB downloads do not stay resident by accident. File-reference refresh is exposed as a low-level file_reference_refresher hook and concurrent refreshes for the same old reference are deduplicated. Client.download_media() also installs a message-backed refresher automatically when the media source is a raw Message or a Media whose raw value is a raw Message; it refetches that message through messages.getMessages or channels.getMessages and retries the part with the refreshed file reference.

Ordinary non-CDN downloads can opt into Telegram’s upload.getFileHashes verification with verify_plain_hashes=True; the default remains off for compatibility and throughput. Hash metadata is validated and cached by effective file-location identity, missing coverage is fetched through the same target-DC media context, and SHA-256 is checked before a part is yielded or submitted to a destination writer. Hash intervals may differ from upload.getFile request boundaries: miniproto assembles and verifies the complete bounded hash interval, then compares and trims the requested overlap. Invalid intervals, gaps, contradictory overlaps, response loops, location changes, and digest mismatches raise MediaIntegrityError with only the non-secret offset and length. CDN verification remains mandatory and unchanged. The disabled path sends no upload.getFileHashes requests.

CDN

When upload.getFile returns upload.fileCdnRedirect, Phase 10 follows upload.getCdnFile, handles upload.cdnFileReuploadNeeded with upload.reuploadCdnFile, and decrypts returned CDN chunks with AES-CTR using the redirect key and IV. Real CDN DC authorization and live edge cases still need gated integration coverage.

Live Tests

The implemented opt-in live check is tests/integration/test_media_live.py. It requires MINIPROTO_INTEGRATION=1, explicit production confirmation through MINIPROTO_REAL_INTEGRATION=1, API credentials, a phone session, MINIPROTO_TEST_UPLOAD_FILE, and optionally MINIPROTO_TEST_DOWNLOAD_PATH; it skips when those external prerequisites are absent. The upload/download byte-comparison check passed in the recorded 2026-08-19 local live run, but every future execution must remain separately labeled from deterministic fake-server and offline release evidence.