Skip to content

WebHTTrack keeps option fields it never sends, and mutes a blank User-Agent - #1422

Merged
xroche merged 2 commits into
masterfrom
fix-1386-webhttrack-optfields
Aug 25, 2026
Merged

WebHTTrack keeps option fields it never sends, and mutes a blank User-Agent#1422
xroche merged 2 commits into
masterfrom
fix-1386-webhttrack-optfields

Conversation

@xroche

@xroche xroche commented Aug 24, 2026

Copy link
Copy Markdown
Owner

The wizard's type-check list and its eight extension/MIME pairs were written to winprofile.ini and read back into the form, but step4.html never put them on the command line, so the crawl never saw them. The browser-id box had the opposite problem: cleared, it emitted --user-agent "", and an empty -F switches the header off instead of falling back to the engine's own value. The command block now emits --check-type=N, one --assume "ext=mime" per filled pair, and nothing at all for a blank browser id. This is engine-side only: the keys already exist in the shared winprofile block, hts_optalias already carries check-type and assume, and winprofile-keys.tsv is untouched.

The second commit fixes a footgun the first one opened and a twin already on master. ${arg:mimeN} sat outside the ${test:extN:} guard, so emptying an extension while leaving its MIME type filled dropped a bare token onto the command line, which the engine reads as options of its own: a MIME field holding -O /some/path silently relocates the whole mirror. ${arg:portprox} has the same shape and predates this PR. Nine sites in all, now wrapped in ${do:if-not-empty:<guard>} the way the proxy scheme already was. htsserver is a local UI and the person filling the form is the person whose crawl it is, so this is a way to lose work rather than a privilege boundary anyone crosses, but emptying one box and not its neighbour is an ordinary thing to do. Test 358 asserted the absence of =<mime> rather than of the value, so it went green on the injecting tree; it now asserts the value is gone and probes the shape directly.

windebug is the original bug mirrored, reaching the engine as --debug-headers and never saved, so it reverts on reload. That one needs a new key in the shared block and WinHTTrack's own spelling has to decide it, so I left it to the httrack-windows side. An extension beginning with a dash still kills the run before any log is written, which is #1179's narrow allowlist rather than anything here: filed as #1425. Test 274 asserted --user-agent "" as the observable for #1186, that a key present in the file beats the wizard's startup default; the property still holds and is now checked on the rendered box, which is what #1186 was about.

Verified against a running htsserver rather than by reading templates: the new test posts browser-shaped forms, starts a real crawl and reads hts-cache/doit.log for the argv the engine got. The --assume half also has to change the mirror, since data.foo is only retyped and parsed once the rule arrives.

Closes #1386

…-Agent

The type-check list and the eight extension/MIME pairs were saved to
winprofile.ini and read back into the form, but step4.html never put them on
the command line. The browser-id box had the opposite problem: cleared, it
emitted --user-agent "", and an empty -F switches the header off rather than
falling back to the engine's own value.

The command block now emits --check-type=N, one --assume "ext=mime" per filled
pair, and nothing for a blank browser id. Engine-side only: the winprofile keys
and the hts_optalias rows already exist.

windebug is the same bug mirrored and is held back, since saving it needs a new
key in the block shared with WinHTTrack.

Closes #1386

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
${arg:mimeN} was emitted whether or not its extension was, so emptying an
extension while leaving the MIME type filled put a bare token on the command
line, which the engine reads as options of its own. ${arg:portprox} has the
same shape on master, guarded by the proxy host rather than by itself.

Nine sites in all, wrapped in ${do:if-not-empty:<guard>} the way the proxy
scheme already is. Test 358 asserted the absence of "=<mime>" rather than of
the value, so it passed on the injecting tree; it now asserts the value is gone
and probes the shape directly.

An extension beginning with a dash still kills the run (#1425).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
@xroche
xroche enabled auto-merge (squash) August 24, 2026 22:03
@xroche
xroche merged commit ecaf54e into master Aug 25, 2026
44 of 45 checks passed
@xroche
xroche deleted the fix-1386-webhttrack-optfields branch August 25, 2026 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebHTTrack saves 17 option fields it never sends, and blanks the User-Agent

1 participant