Intermediate snapshots are not synced when using ZREP_TAG / -t

[t-oster]

Hi,

I just tested the following:

• zrep -i mypool/test myremote remotepool/test
• zfs snap mypool/test@custom
• zrep -S mypool/test
• zrep -t zrep-customtag -i mypool/test myremote remotepool/test2
• zfs snap mypool/test@custom2
• zrep -S mypool/test
• zrep -t zrep-customtag -S mypool/test

The remotepool/test contains @Custom and @custom2, aswell as zrep and zrep-customtag snapshots as expected. However remotepool/test2 only contains the @zrep-customtag snapshots, but neither @Custom nor @zrep snapshots.

Is that expected behavior? I'd expect at least all intermediate snapshots to be synced like it is without the -t parameter.

[ppbrown]

interesting. that is not something i usually concern myself with since that is behaviour of underlying zfs implementation. One has to deal with what one is given :) if you find a zfs flag that triggers behaviour more like you want, please let me know, and i can figure out the best way to allow it in zrep. possibly there may be an extra flag usable at initial (second init) snapshot creation time.

…

On Tue, May 4, 2021 at 1:26 AM Thomas Oster ***@***.***> wrote: Hi, I just tested the following: - zrep -i mypool/test myremote remotepool/test - zfs snap ***@***.*** - zrep -S mypool/test - zrep -t zrep-customtag -i mypool/test myremote remotepool/test2 - zfs snap ***@***.*** - zrep -S mypool/test - zrep -t zrep-customtag -S mypool/test The remotepool/test contains @Custom <https://github.com/Custom> and @custom2, aswell as zrep and zrep-customtag snapshots as expected. However remotepool/test2 only contains the @zrep-customtag snapshots, but neither @Custom <https://github.com/Custom> nor @zrep <https://github.com/zrep> snapshots. Is that expected behavior? I'd expect at least all intermediate snapshots to be synced like it is without the -t parameter. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#168>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANEV6MYOJFWJ6NJIPXORW3TL6VSBANCNFSM44CIANBA> .

[t-oster]

well... I guess this is inteded, as I found in the source:

# There are two types of zfs incremental snapshots.
#   The default incremental flag is -I. 
#   UNLESS you set ZREPTAG to something other than zrep, in which case
#   you will have multiple zrep snapshot names probably going to different
#   places, and expiration wont work properly on the remote sides
#   So we will autochange incremental type to -i. .. unless you explicitly 
#   set an override value for INC_FLAG in either case.
#   Probably should have named this INCR_FLAG, but it's in use now

so you are auto-changing -I to -i as soon as I set a tag and that's why the intermediate snapshots are ignored.

[ajshea]

Setting the flag does not seem to change the behavior. In my case I DO want all the daily intermediate snapshots on the external backup for offsite/offline storage.

[ppbrown]

Hmm.
Do you think you may be "setting" it, but forgetting to actually export INC_FLAG ?

Or, Do you see any potential bug in zrep that may not be allowing it through?

[ajshea]

This is zrep v2.0.2 (latest release from master) with #!/bin/bash on FreeBSD 13.3 XigmaNAS and bash 5.2.26. ksh v93u from AST is also installed (default shell is tcsh v6.22.04).

"export" doesn't exist in FreeBSD, so I'm using set ZREP_INC_FLAG=-I before running the zrep command. I also included ZREP_INC_FLAG=-I in /etc/defaults/zrep (the path in FreeBSD 13x is /etc/defaults/ plural so I modified zrep to look there instead). Still no joy.

I tried uncommenting the ZREP_INC_FLAG=-I in the opening comments (line 122).

I also tried forcing the flag in line 2711:

         # In this case, DO NOT send all intermediate snapshots for replication
         ZREP_INC_FLAG=${ZREP_INC_FLAG:-"-I"}

For the initial "init" ,it still only sends the current zrep snapshot, not all snapshots.

I did not try the version from #197, I could try that.

[ppbrown]

Your comment of "export doesnt exist in FreeBSD" does not make sense.
It isnt a feature of the OS. It's a feature of the shell.
Originally zrep was written in ksh.
newest versions are written in bash.
BOTH shells should support "export".

That being said:

"For the initial "init" ,it still only sends the current zrep snapshot, not all snapshots."

I dont think that was what the original bug report was about.
As I understood it, the bug was that intermediate snapshots were not created on followup sync.
Whereas you seem to be concerned about it on init?
That is a completely diffferent use case, and probably should have its own seperate bug. Feel free to open one.
Please document your use case from scratch in the new bug.

[ajshea]

Sorry, I meant export isn't supported in tcsh, maybe I shouldn't be launching zrep from tcsh?

I guess I assumed that the historical snapshots got rolled up in the init snapshot zfs send and not just the present state. Will continue in a new bug report, #213.

Confirming that he intermediate snapshot does get included with the zrep version from #197 and with the version from master (release 2.02) when ZREP_INC_FLAG=-I is set in /etc/defaults/zrep.