WebHTTrack keeps option fields it never sends, and mutes a blank User-Agent - #1422
Merged
Conversation
…-Agent The type-check list and the eight extension/MIME pairs were saved to winprofile.ini and read back into the form, but step4.html never put them on the command line. The browser-id box had the opposite problem: cleared, it emitted --user-agent "", and an empty -F switches the header off rather than falling back to the engine's own value. The command block now emits --check-type=N, one --assume "ext=mime" per filled pair, and nothing for a blank browser id. Engine-side only: the winprofile keys and the hts_optalias rows already exist. windebug is the same bug mirrored and is held back, since saving it needs a new key in the block shared with WinHTTrack. Closes #1386 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
${arg:mimeN} was emitted whether or not its extension was, so emptying an
extension while leaving the MIME type filled put a bare token on the command
line, which the engine reads as options of its own. ${arg:portprox} has the
same shape on master, guarded by the proxy host rather than by itself.
Nine sites in all, wrapped in ${do:if-not-empty:<guard>} the way the proxy
scheme already is. Test 358 asserted the absence of "=<mime>" rather than of
the value, so it passed on the injecting tree; it now asserts the value is gone
and probes the shape directly.
An extension beginning with a dash still kills the run (#1425).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The wizard's type-check list and its eight extension/MIME pairs were written to
winprofile.iniand read back into the form, butstep4.htmlnever put them on the command line, so the crawl never saw them. The browser-id box had the opposite problem: cleared, it emitted--user-agent "", and an empty-Fswitches the header off instead of falling back to the engine's own value. The command block now emits--check-type=N, one--assume "ext=mime"per filled pair, and nothing at all for a blank browser id. This is engine-side only: the keys already exist in the shared winprofile block,hts_optaliasalready carriescheck-typeandassume, andwinprofile-keys.tsvis untouched.The second commit fixes a footgun the first one opened and a twin already on master.
${arg:mimeN}sat outside the${test:extN:}guard, so emptying an extension while leaving its MIME type filled dropped a bare token onto the command line, which the engine reads as options of its own: a MIME field holding-O /some/pathsilently relocates the whole mirror.${arg:portprox}has the same shape and predates this PR. Nine sites in all, now wrapped in${do:if-not-empty:<guard>}the way the proxy scheme already was. htsserver is a local UI and the person filling the form is the person whose crawl it is, so this is a way to lose work rather than a privilege boundary anyone crosses, but emptying one box and not its neighbour is an ordinary thing to do. Test 358 asserted the absence of=<mime>rather than of the value, so it went green on the injecting tree; it now asserts the value is gone and probes the shape directly.windebugis the original bug mirrored, reaching the engine as--debug-headersand never saved, so it reverts on reload. That one needs a new key in the shared block and WinHTTrack's own spelling has to decide it, so I left it to the httrack-windows side. An extension beginning with a dash still kills the run before any log is written, which is #1179's narrow allowlist rather than anything here: filed as #1425. Test 274 asserted--user-agent ""as the observable for #1186, that a key present in the file beats the wizard's startup default; the property still holds and is now checked on the rendered box, which is what #1186 was about.Verified against a running htsserver rather than by reading templates: the new test posts browser-shaped forms, starts a real crawl and reads
hts-cache/doit.logfor the argv the engine got. The--assumehalf also has to change the mirror, sincedata.foois only retyped and parsed once the rule arrives.Closes #1386