Commands to add to mmcloud based docker image

[gaow]

Let' me start it. System packages:

apt-get install -y curl

Please reply to this thread add additional lines.

[gaow]

For ANNOVAR, We cannot ship it in any containers because of license issues. What we have to do is to upload a version of it to our S3 bucket and edit our entrypoint script to sync that to our VM on the fly, and set the correct permissions etc.

@lamanda0227 @dianacornejo that will involve some trial and error on your end, then you come up with the correct command to add to:

https://github.com/statgenetics/statgen-courses/blob/master/setup/MMCloud/jn_entrypoint.sh

Also for notebooks / files that are missing from this repo, such as the ngs_qc_annotation related resources, @lamanda0227 please add to the file above and PR.

[lamanda0227]

Need to install

1. fastlmm (Dan)
2. r-corrplot - https://anaconda.org/conda-forge/r-corrplot
3. hudson - https://anaconda.org/dnachun/r-hudson
4. plink.multivariate - https://genepi.qimr.edu.au/staff/manuelF/multivariate/plink.multivariate.zip
5. annovar - http://www.openbioinformatics.org/annovar/download/0wgxR2rIVP/annovar.latest.tar.gz

I am only putting the link here today, will work on the actual command on how to install them tomorrow.

Also I wonder if we currently have any environment for now? I was not able to activate/find any. Currently only python_libs and r_libs are available

For the sos kernel, I tried to download it using pixi, it seems that it can be downloaded but does not show up.

[gaow]

Also I wonder if we currently have any environment for now?

From the user's prospective, with pixi we should no longer need any environments! Dan also helped me updating our lab wiki, https://wanggroup.org/orientation/jupyter-setup as you can see no environments anymore. Although we need some "pioneers" to test it before i ask the entire lab to do so ...

Currently only python_libs and r_libs are available

This is temporary as well and Dan hacked it such that we don't notice that they are environments. Down the road with a feature already proposed in pixi we will be able to get rid of micromamba completely. Currently is a pixi + micromamba hybrid

For the sos kernel, I tried to download it using pixi, it seems that it can be downloaded but does not show up.

My next patch should solve it.

[gaow]

@lamanda0227 as a reminder we need to upload the data also to synapse.org i think i sent you the link of it before if you search on slack. You can ask Ru the login information it's the same account that my lab uses. Make the files publicly avaiable by setting permission on synapse to something like "public can view/download".

[lamanda0227]

Sure. I am going to do that after I reorganize the data in the S3 bucket

[gaow]

2. Clumping - no handout

it's here. Please download it on the fly like ldpred2 https://github.com/cumc/bioworkflows/blob/master/GWAS/LD_Clumping.ipynb

Please add it to jn_entrypoint.sh

1. Remove CASSI section from the handout pdf?

Please discuss with Suzanne. We definitely can install CASSI if there is a need so it's just a matter of if there is a need.

1. ldpred pipeline returns error![image-20240410160547879](/Users/tianyi/Library/Application Support/typora-user-images/image-20240410160547879.png)

Can't see any images; but please try with updated VM and see.

1. Dataset is different than the handout

Ask Suzanne

1. GWAS-PARTII_command.txt -> `plink --noweb --file GWAS_clean4 --genome --mds-plot 10` this line of command outputs an error

Drop --noweb and try again?

1. Pipeline returns error: ![image-20240410160515489](/Users/tianyi/Library/Application Support/typora-user-images/image-20240410160515489.png)

Can't see any image; but please try with updated VM and see.

[gaow]

A reminder to myself: regarding the request from Suzanne to still support offline applications on student laptops --- @lamanda0227 let's talk in person about it after all tests are completed. I may have a simpler solution for that compared to completely falling back to our previous solution, thus avoid maintaining two approaches to do the same thing.

[gaow]

Basically this involves a small change in our current approach, plus a bit of a testing but most importantly, updating our previous versions of documentations.

[lamanda0227]

@gaow Just updated the ngs_qc_annotation notebook. But I have a question regarding downloading the annovar pipeline on the fly - I modified the original annovar pipeline such that it only uses 3 databases to annotate. I wonder how should we deal with this one?

[lamanda0227]

Second round of testing

General comments

1. Still not able to use sos kernel in jupyter notebook, but runs fine in bash consoles/kernels. It's just when opening the notebook, sos kernel will be connecting for a while and then turn into 'no kernel'
2. Edit the entrypoint script to copy github raw content, currently not doing the right thing... @lamanda0227

EXERCISE

Annovar

Generates this error

NOTICE: Running with system command <convert2annovar.pl  -includeinfo -allsample -withfreq -format vcf4 APOC3.vcf > APOC3_Gene.vcf.avinput>
sh: line 1: convert2annovar.pl: command not found
Error running system command: <convert2annovar.pl  -includeinfo -allsample -withfreq -format vcf4 APOC3.vcf > APOC3_Gene.vcf.avinput>

May need to chmod +x all .pl's into executables and move them to bin @yiweizh-memverge

Epistasis

Confirmed with Suzanne, she wanted us to try installing CASSI before change/removing the exercise.
Link for downloads: https://www.staff.ncl.ac.uk/richard.howey/cassi/downloads.html
I wonder if we want to do the same thing for CASSI like annovar? @gaow @yiweizh-memverge

LDpred2

Missing R packages: gplots with dependencies bitops gtools caTools @yiweizh-memverge

Pleiotropy

1. plink.multivariate seem to run but does not generate anything, no help messages printed out when running plink.multivariate --help
2. Remove pleiotropy_commands_2021.txt @lamanda0227

ngs_qc_annotation

Fix the db and entry point script @lamanda0227

plink-r-nothnagel

Command

plink --noweb --file GWAS_clean4 --genome --mds-plot 10

produces an error saying --mds-plot must be used with --cluster together. However, after adding --cluster to the command, it does not produce the output file required by downstream R analysis. -> I don't think we can solve this issue

Regenie

This exercise is still quite slow to run, and currently the first step has been running for 22 mins and not finished
Move all data into regenie_statgen_mwe folder @lamanda0227

[yiweizh-memverge]

Re: Annovar:

• Should be an easy fix

Re: LDPred2:
@gaow Is it okay just to put those into r_libs.yml? And just to confirm the packages below:

• r-bitops: https://anaconda.org/conda-forge/r-bitops or https://anaconda.org/r/r-bitops Any version requirements?
• r-gplots: https://anaconda.org/conda-forge/r-gplots or https://anaconda.org/r/r-gplots or https://anaconda.org/bioconda/r-gplots ? Any version requirements?
• r-gtools: https://anaconda.org/conda-forge/r-gtools or https://anaconda.org/r/r-gtools Any version requirements?
• r-caTools: https://anaconda.org/conda-forge/r-catools or https://anaconda.org/r/r-catools Any version requirements?

[gaow]

@lamanda0227 okay just coming back on this. In my view these are all addressible and will be addressed in the next iteration although a couple of them you need Suzanne's input.

Please test another round once @yiweizh-memverge has the image out. We want things to work 100%. Thanks!

[gaow]

@yiweizh-memverge @lamanda0227 because I was told that we must strictly ensure that the old way of running the tutorials (on individual laptops) also works, I am taking the easiest approach here: instead of trying to make the repo backwards compatible, I have reverted master to the state before mmcloud, because many of our existing docker images has paths and data hard-coded to work with the previous structure of the repo. So we should not change the repo to make sure it works 100% with the old setup.

For our course on mmcloud I therefore created a new repo here: https://github.com/cumc/handson-tutorials

A couple of changes relevant to @yiweizh-memverge and @lamanda0227 :

1. This file has been changed already, https://github.com/cumc/handson-tutorials/blob/main/setup/course_entrypoint.sh including the filename. What I need your help with are to change the paths, s3://opcenter-bucket-ada686a0-ccdb-11ee-b922-02ebafc2e5cf/tutorial_data without annovar_software and also s3://opcenter-bucket-ada686a0-ccdb-11ee-b922-02ebafc2e5cf/annovar_software/ . @lamanda0227 could you rename the folders accordingly on S3? @yiweizh-memverge you need to test it out see if it works and fix issues
2. @yiweizh-memverge your documentation on how to set it up goes here: https://github.com/cumc/handson-tutorials/blob/main/setup/README.md for developers. The documentation how to run the tutorials go here: https://github.com/cumc/handson-tutorials/blob/main/getting-started-mmcloud.md

[gaow]

please double check particularly this line https://github.com/cumc/handson-tutorials/blob/main/setup/course_entrypoint.sh#L32 to make sure it works as intended.

[yiweizh-memverge]

For this doc: https://github.com/cumc/handson-tutorials/blob/main/getting-started-mmcloud.md, is this for how to deploy the instances on MMC?

[gaow]

@yiweizh-memverge this note is actually for users to read. So this is when you pretend yourself as a user / student who is new to the class on day 1, how are they going to use the exercise. I guess maybe it is better that @lamanda0227 does this later. ....

[gaow]

@yiweizh-memverge this one https://github.com/cumc/handson-tutorials/blob/main/setup/README.md is to pretend yourself as a developer role, but who has not done any of it before, let's say anyone with opcenter and AWS bucket already configured, but no setup for the course and wants to do exactly do what you did to set it up, what are the steps.

[lamanda0227]

Third round of testing

Basic_plink_r_nothnagel

• S3 - move data plink-r-nothnagel/multtest/p.value.R to basic_plink_r_nothnagel
• Package - Need multtest package in R

Clumping - no commands to run

Epistasis

• Update LICENSE to Heather Cordell

FastLMM-GCTA ✅

finempping ✅

LD-pred2 ✅

Multivariate finemapping ✅

ngs_qc_annotation

• S3 - Generate first step result and upload to s3 folder ngs_qc_annotation/output/bcftools_qc/

• S3 - Remove the fasta file and db

• The annovar pipeline generates this error (solved by adding --container_annovar '')

  ERROR: annovar_2 (id=e67dcb203d240e23) returns an error.
  ERROR: [annovar_2]: [0]: 
  ---------------------------------------------------------------------------
  RuntimeError                              Traceback (most recent call last)
  script_5552982495529399589 in <module>
  ----> bash(fr"""#do not add -intronhgvs as option -> writes cDNA variants as HGVS but creates issues (+2 splice site reported only)
        #-nastring . can only be . for VCF files
        #regsnpintron might cause shifted lines (be carefull using)
        table_annovar.pl \
  
  RuntimeError: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
  

pleiotropy

• Git - Delete pleiotropy_commands_2021.txt from GitHub
• Fix the bugs in the pleiotropy_commands.txt

plink

• Git - Remove ~/work/from the PLINK_GWAS_commands.txt
• Git - Remove the plink_qc_pcafolder, it contains the same content

plink-r-nothnagel

• S3 - Move the data to the correct location

popgen-nothnagel

• Git - remove the popgen folder

regenie

• regenie pipeline generates this warning but the pipeline run and can finish

  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:7: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:7: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:7: SyntaxWarning: invalid escape sequence '\s'
  <string>:3: SyntaxWarning: invalid escape sequence '\s'
  <string>:7: SyntaxWarning: invalid escape sequence '\s'
  .....
  

regression_nothnagel

• Git - remove the original regression folder, it is a duplicate
• S3 - move the plink-r-nothnagel/regression.I/* into regression_nothnagel

Staten_equations

• The BLUP step has this line:

  if(is.null(out_type)) out_type = ''
  

  and the error is

  Error in eval(expr, envir, enclos): object 'out_type' not found
  Traceback:
  

  If I change this line to

  out_type = ''
  

  this warning, but runs

  Warning message:
  “'rgl::rgl.viewpoint' is deprecated.
  Use 'view3d' instead.
  See help("Deprecated")”
  Warning message:
  “'rgl::rgl.lines' is deprecated.
  Use 'segments3d' instead.
  See help("Deprecated")”
  Warning message:
  “'rgl::rgl.lines' is deprecated.
  Use 'segments3d' instead.
  See help("Deprecated")”
  ......
  Warning message:
  “Combining widgets requires the 'manipulateWidget' package.”
  Warning message:
  “Combining widgets requires the 'manipulateWidget' package.”
  

• The last third step with generating a heatmap, resolution = 300 taking too long, resolution = 30 works fine.

mendelian_randomization

• Git - make a mr folder with code folder, upload the R code Andy provided (no data needed)
• Failed to installMRInstruments, so did not test this exercise