DAOS-18694 tests: test for container recovery when inc-reint

[Nasf-Fan]

Related C test logic has already been there, only add related python interface to enable CI test.

Skip-nlt: true
Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: cont_recov

Steps for the author:

• Commit message follows the guidelines.
• Appropriate Features or Test-tag pragmas were used.
• Appropriate Functional Test Stages were run.
• At least two positive code reviews including at least one code owner from each category referenced in the PR.
• Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

• Gatekeeper requested (daos-gatekeeper added as a reviewer).

[github-actions]

Ticket title is 'Enable test for container recovery when incremental reintegration'
Status is 'In Progress'
https://daosio.atlassian.net/browse/DAOS-18694

[daosbuild3]

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17938/1/execution/node/736/log

[Nasf-Fan]

Pass all specified CI tests

[phender]

Historically we've placed tests running daos_test in the ftest/daos_test directory - like ftest/daos_test/rebuild.py. If we plan to have multiple reintegration tests does it make sense to have a ftest/daos_test/reintegration.py test file/yaml?

[phender]

Historically we've placed tests running daos_test in the ftest/daos_test directory - like ftest/daos_test/rebuild.py. If we plan to have multiple reintegration tests does it make sense to have a ftest/daos_test/reintegration.py test file/yaml?

I also do like having a rebuild related test in the ftest/rebuild directory. It would just be good to have consistency - something outside the scope of this PR.

[daltonbohning]

FYI we should add the daos_test and probably rebuild tags as well
But I have another PR #17975 where I will handle this so you don't need to repush for just this

Suggested change

	:avocado: tags=inc_reint,cont_recov
	:avocado: tags=daos_test,rebuild,inc_reint,cont_recov

[Nasf-Fan]

Historically we've placed tests running daos_test in the ftest/daos_test directory - like ftest/daos_test/rebuild.py. If we plan to have multiple reintegration tests does it make sense to have a ftest/daos_test/reintegration.py test file/yaml?

Currently, we support full reintegration, means reintegrate rank/target from scratch. We have related tests cases, but I do not know why they are not under reintegration.py. I am not the maintainer for rebuild/reintegration, but I guess that it may because the logic for reintegration and rebuild are quite similar, then they were misc together at the beginning.

On the other hand, full reintegration may cause too much overhead for a large system, then we plan to support incremental reintegration in 3.0 release. So I made patch to handle container recovery for the coming incremental reintegration that will be done by @liuxuezhao in future. These test cases in inc_reint.xxx are just for such purpose. I am not sure @liuxuezhao 's plan about how to organize related test cases for incremental reintegration since the main part is not completed yet. But the logic for handling container recovery has already been landed to master. So I prefer to enable these test cases in inc_reint.xxx relative independently, and waiting for @liuxuezhao 's main part to be completed, and at that time, @liuxuezhao may re-organize related test cases together.