I checked what make loaddata actually pulls before pulling it. Your bucket reconciles to the byte against the LOG it ships.

hf buckets ls on cc-news-cdx-server-storage gives 1,123 objects, 1,111 of them .sst, 75,532,239,021 bytes. Take out the 12 non-sst files and the SSTs come to 75,516,444,342.

Your own LOG, same bucket. At open on 08-01 the picker prints files[3 0 0 0 61 159 885], so 1,108 live files, and the delete scheduler prints total_size 75,066,314,049. Three flushes land after that: 5851 at 187,953,067, 5854 at 194,745,281, 5856 at 67,431,945.

1,108 + 3 = 1,111. And 75,066,314,049 + 450,130,293 = 75,516,444,342.

Same number both ways. So the sync is not shipping dead SSTs, which is the thing I went looking for and did not find.

What it does ship is 15,794,679 bytes the DB no longer owns. CURRENT reads MANIFEST-005845, so MANIFEST-000005 is a dead 1.7 MB, OPTIONS-000007 with it, LOG.old at 13.6 MB, and 005839.log, which your LOG records deleting at 16:46:03.895773 that same morning. It is still in the bucket, dated 07-23 12:55:59.

0.02%, so not worth a fix on its own. It is worth knowing why it is there. hf sync defaults to --no-delete, so the bucket is additive, and L6 is 885 files and 55.77 GiB. The first compaction that retires part of that leaves the retired files behind, and the puller downloads them.

The other thing the LOG says is about the state it froze in. The DB closed 41 seconds into a compaction that never landed. Job 3 logs compaction_finished, 4 output files, 206,687,332 bytes to L3, and the next lines delete 5848, 5849, 5852 and 5857. lsm_state still reads [6, 0, 0, 0, 61, 159, 885] with L3 empty, and the inputs 5818 and 5770 are still in the bucket. So what ships is pre-compaction: six sorted runs in L0, max score 1.50.

Your compose handles that. :ro mount plus ROCKS_SHADOW means a puller never writes into it.

The README's Local Setup path is the one I would check. Does uv run cdx-rocks "$ROCKS_READONLY" "$ROCKS_SHADOW" open read-only too, or does the first local run re-do that compaction on someone else's disk?

The --delete fix lands on the bucket, but the August refresh breaks twice before it gets there. I ran both.

First is make loaddata. rocksdb_index is a directory prerequisite, so make calls it satisfied the moment it exists. Second run, a month later, new data in the bucket:

make: Nothing to be done for 'loaddata'.

Exit 0. No sync, no output, and the server comes back up on the July index.

Second is the guard. You said setup_shadow runs if the shadow directory is empty. The code checks is_dir():

if not Path(ROCKS_SHADOW).is_dir():
    setup_shadow(Path(ROCKS_READONLY), Path(ROCKS_SHADOW))

Your setup_shadow is idempotent, it unlinks and relinks every entry. The guard is what never lets it run twice. And ROCKS_SHADOW is /app/rocksdb, which lives in the container layer and not a volume, so it survives restart: unless-stopped.

I ran your setup_shadow and your read-only open against a real rocksdict DB. July, 500 keys, sync, boot. Then indexed 500 more, synced again with the --no-delete default, restarted:

boot 1   setup_shadow ran: True    9 entries   OPEN OK, 500 keys
boot 2   setup_shadow ran: False   9 entries   OPEN FAILED
         IO error: No such file or directory: MANIFEST-000013
boot 3   setup_shadow ran: True   14 entries   OPEN OK, 1000 keys

boot 3 is that same August data with the shadow wiped first.

Why it is fatal and not merely stale: CURRENT keeps its filename, so the additive sync overwrites it through the symlink, and it now names MANIFEST-000013. Every other August file arrived under a new number with no link pointing at it. So --no-delete hides the staleness on every file except the one the open reads first.

A CURRENT content check instead of is_dir() refreshes it. Or put the shadow somewhere the layer cannot keep, since rebuilding it is only symlinks.

Does the ccnget side pin an index version, or does it trust whatever /lookup happens to have mounted?

Then the failure ccnget cannot see is not a 404. It is a 200.

The catalog is a separate artifact from the index. ID_TO_PATH is built positionally at boot, enumerate(text_stream, start=1), and the RocksDB value stores only that position: struct.unpack("!HQI", value). So a capture resolves through whatever catalog the process happened to load.

I ran your app/main.py against a real rocksdict with two captures, ids 2 and 5, and booted it twice with different catalogs. Fresh shadow both times, so this is not the symlink thing:

CONTROL  catalog 5 paths (matches index)
  ts=20260601120000  offset=111111  warc_path=crawl-data/.../00002.warc.gz
  ts=20260731193544  offset=999999  warc_path=crawl-data/.../00005.warc.gz

SKEW     catalog 3 paths (one refresh behind)
  ts=20260601120000  offset=111111  warc_path=crawl-data/.../00002.warc.gz
  ts=20260731193544  offset=999999  warc_path=PATH_NOT_FOUND

Both are HTTP 200. Both validate against LookupResponse. The 404 only fires on if not captures, and captures is non-empty, so PATH_NOT_FOUND ships to the client as a warc_path with a real offset and length beside it.

What produces the skew is the same Makefile line as before, on its other leg:

loaddata: rocksdb_index all_warc_paths.txt.zst

Two existence-only targets. So the natural fix for the staleness, delete the index and re-sync, refreshes exactly one of them:

$ rm -rf rocksdb_index && make -n loaddata
uvx hf sync hf://buckets/.../cc-news-cdx-server-storage rocksdb_index

New index, July catalog. Deleting both gets both.

The version token you say ccnget lacks is already computed. Your lifespan logs Catalog loaded (51,101 paths) and then never returns it. Put that count in the response and a client can pin it.

Worth knowing what that number is doing, though. I pulled all_warc_paths.txt.zst: 51,101 paths, and !H addresses 65,535. CC-NEWS added 459/month over the last twelve and 366/month over the last six, so the id space runs out in 2.6 to 3.3 years.

Does the builder pack ids from the same catalog file the server loads, or from a listing it takes at build time?

Your catalog is append-only, and I checked instead of taking your word. It bounds my last message.

all_warc_paths.txt.zst at rev fabe94d ("corrected sort order", 07-19) is 50,958 lines. At c603b13 (08-01) it is 51,101. The first 50,958 lines are byte-identical, md5 0d9c6c5063fa5b72c6c2069bec6067b0 both times. The July refresh appended 143 and moved nothing.

So a stale catalog cannot remap an old id onto a wrong path. It can only run off its own end. PATH_NOT_FOUND is the only wrong answer available, never a real path pointing at the wrong file. That is a stronger property than I gave you credit for.

I checked the join too, since both builders key name_to_id on os.path.basename while the catalog stores full paths. All 51,101 basenames are distinct, so the dict loses nothing. Clean.

The 65,535 ceiling is worse than needing a new format, though. It is not a crash.

struct.error subclasses Exception, and the pack sits inside except Exception: continue in both rocksdb_from_cdxj.py and rocksdb_monthly.py. I ran your loop body on 5 real cdxj lines from cc-news_2016_08, changing only the id:

id=51,101   5 records written   "Update complete: Added 5 records."
id=65,535   5 records written   "Update complete: Added 5 records."
id=65,536   0 records written   "No new records added (file empty or no matches)."
            5 swallowed: 'H' format requires 0 <= number <= 65535

The counter only counts successes. So the first month past the ceiling reports as an empty input file, not an overflow. Every record of every WARC above catalog line 65,535 disappears quietly, and the operator reads a sentence that means "nothing to do here."

On which field is oversized, I measured, and it is not the one you named.

8,141,383 records, the oldest month and the newest:

Neither month's offset ever reaches 2^30. The 2016 max stops 10,776 bytes short of it. Both length maxima stop within about 500 bytes of 1 MiB. Ten years apart, so those are producer-side caps, not my sample.

length wastes 12 bits. offset wastes 34 and nobody has flagged it. The field that breaks is the only one too small. !III is 12 bytes against your 14 and retires the ceiling.

Timing comes out of your own catalog. 51,101 of 65,535 is 78.0%, so 14,434 left. The last six months run 353 to 394, mean 366, so about 40 months. But your rate is not flat. 2025-12 alone was 616, and your all-time high is 745 in 2023-03. At that rate it is 19 months.

Cheap guard in the meantime: refuse to boot if the catalog exceeds 65,535 lines, and log len(name_to_id) against the enumerate count. Does anything downstream compare records-written to cdxj lines-read? If not, the month it happens just looks like a quiet month.

Your value is 14 bytes on disk, not 16. The ! turns padding off.

struct.calcsize("!HQI") is 14. The ! prefix selects standard sizes with no alignment, so there is nothing to pad. VALUE_SIZE at app/main.py:37 already evaluates to 14 today. The comment two lines above it says 16.

Your own two files disagree, and the builder is the one that is right:

rocksdb_from_cdxj.py:112   # Pack into our universal 14-byte footprint:
app/main.py:31             # Each value is a 16-byte big-endian struct:
app/main.py:35             # Total: 2 + 8 + 4 = 14 bytes (struct alignment pads to 16)

Nothing is broken by it. VALUE_SIZE appears exactly once in the whole repo, at its own definition, and the unpack at :143 reads the format rather than the size. So it is a stale comment, not a data bug. It is just the comment the proposal was reasoning from.

Which makes !IQI a real +2 bytes, not a free move into space you had already paid for.

You are still right that it does not change the db, so I built it instead of arguing the arithmetic. cc-news_2016_08, 299,609 records after your inline dedup, your options: raw_mode, zstd, universal compaction, 512MB write buffer. Compacted each one, then summed SST bytes.

!IQI adds 599,218 raw bytes and 56,378 of them reach disk. 9.4% pass-through. The two bytes are the high half of a 16-bit id widened to 32, so they are zeros, and zstd charges almost nothing for them.

So the size question answers itself, and it answers itself the same way for every option. Mean key in that month is 105.5 bytes against a 14-byte value, so the value is 11.7% of the raw kv bytes. Nothing you can do inside three integer fields moves this file.

That leaves headroom as the only axis that decides it. And on headroom the winner is also the smaller one.

The offset field is why. Max offset in 2016_08 is 1,073,731,048, which stops 10,776 bytes short of 2^30. 2026_07 lands in the same place. Ten years apart, both stopping just under 1 GiB, is a producer-side cap on WARC size and not an artifact of my sample. uint32 leaves 4x over a ceiling that has held for the life of the dataset. uint64 is buying range against something nobody has been able to cross.

So !III retires the 65,535 id ceiling, keeps 4x on offset, and gives back 1.05%. !IQI retires the same ceiling and costs 0.45%. Both are fine. Only one of them is smaller than what you ship now.

One thing makes either change cheaper than I expected. I grepped ccnget for struct and got zero hits. The client reads offset and length out of your JSON already unpacked, so the 14 bytes never leave the server and nobody downstream is parsing them.

Which puts the whole cost in the rebuild. What was the wall clock on the last full cdxj-to-rocksdb pass?

Your port worry has an easy answer, and the index-version worry has a date on it.

Port first. The Space is already on 7860 and that is the HF default, so nothing collides. The public URL has no port at all: https://brian-learns-cc-news-cdx-server.hf.space/lookup answers and / redirects to /docs. The port only exists inside the container and on a local docker run. Moving off it means adding app_port: to the README frontmatter, which is one more pair of things to keep in sync. I would leave it.

Before the rest I pulled your index end to end. Asked the Space for bbc.com/news, took the row it gave back:

crawl-data/CC-NEWS/2018/04/CC-NEWS-20180408051913-00192.warc.gz  offset 677849923  length 56405

Range-fetched exactly those bytes from data.commoncrawl.org and gunzipped:

WARC-Target-URI: http://www.bbc.com/news
WARC-Date: 2018-04-08T09:16:22Z

Matching the 20180408091622 you returned. The id-to-path mapping is correct.

Now the id field, because it is the one with a deadline. all_warc_paths.txt.zst is 51,101 lines, 78.0% of 65,535. CC-NEWS adds 365.5 files a month over the trailing six, 459 over the trailing twelve and twenty-four. So uint16 runs out in 31 to 40 months, somewhere in 2029.

Which sharpens the argument I made badly last time. !IQI and !III both retire the field that has a date. The extra bytes !IQI spends go to the offset, and offset has no date, since the max in both 2016_08 and 2026_07 stops just under 2^30 and uint32 already carries 4x. Same fix on the field that matters, either way, so !IQI is a defensible call.

One thing about the catalog itself that I did not expect. It is not append-only. All 51,101 lines are exactly sorted lexicographically and the embedded timestamps are monotonic, so it is regenerated by sorting. A backfilled WARC in an old month is an insert, not an append. That 2018 file is id 2339, so an insert there shifts 48,762 ids, 95.4% of them. And nothing binds the two artifacts: the catalog is COPYd into the image at build time, the rocksdb mounts at /data, and they version independently.

I checked how loudly that fails rather than guessing. Refetched the same offset and length under id 2340, the next path over:

gzip: Not a gzipped file (b')/')

So drift fails at decompression, not with a plausible wrong article. That is the good outcome. It is still undiagnosable from the client, which is the same shape as the thing you said you did not want ccnget to carry.

Which is where the two indexes come in, and it is bigger than versions.

30 of 75 is 40%. By your own cdxj bytes, 115.28 GB across 120 months, a recency-ordered 40% starts at 2022-10. 46 months kept, 74 dropped, and 50.5% of your WARC ids sit before that line.

I measured what the client sees using your own timestamp seek. Per URL, exact=true limit=1, then exact=true at=20221001 limit=1.

Of 17 news front pages, 6 have an exact-SURT capture. Five of those six have nothing at or after 2022-10-01:

www.bbc.com/news                first 20180408091622   after cutoff: none
www.reuters.com/world           first 20210722030958   after cutoff: none
www.cnn.com/politics            first 20170101074548   after cutoff: none
www.thehindu.com/news           first 20210203142541   after cutoff: none
www.lemonde.fr/international    first 20210814122104   after cutoff: none
timesofindia.../world           first 20260213115325   survives

n is 6, so take the five-of-six as a direction and not a rate. The mechanism is the solid part.

Your newest commit points the same way. "make not 404 on no results" means a miss is now a 200 with an empty body, which is the right call for a client, and it also means the two kinds of miss look more alike than ever. openapi.json has one path. LookupResponse is seven fields: query_url, surt_prefix, exact_match, at_timestamp, total_results, limit, results. None of them names the index. I ran a hit and a certain miss against the live Space just now, and the miss is total_results 0, results empty, with nothing else in the object to tell a subset miss from a real one.

So the client cannot stay dumb by being told nothing. It stays dumb if the server says what it covers. Earliest and latest timestamp, or a coverage label, in the response or on an /about route. Then ccnget never learns about index versions, it just repeats what it was told.

Would you put coverage in LookupResponse itself, or keep /lookup lean and add a route?