Stuck at plan(multisession)

[fnavarro94]

I am having trouble running the command

plan(multisession, workers = 2)

when I do so I get the following warnings and the code never runs past that command. Help much appreciated!

  running command '"C:/PROGRA~1/R/R-41~1.2/bin/x64/Rscript" -e "try(suppressWarnings(cat(Sys.getpid(),file=\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=1.parallelly.parent=21780.55145ced6e18.pid\")), silent = TRUE)" -e "file.exists(\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=1.parallelly.parent=21780.55145ced6e18.pid\")"' had status 1
Warning in system(test_cmd, intern = TRUE, input = input) :
  running command '"C:/PROGRA~1/R/R-41~1.2/bin/x64/Rscript" -e "try(suppressWarnings(cat(Sys.getpid(),file=\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=2.parallelly.parent=21780.551494a13f1.pid\")), silent = TRUE)" -e "file.exists(\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=2.parallelly.parent=21780.551494a13f1.pid\")"' had status 1

 version  R version 4.1.2 (2021-11-01)
 os       Windows 10 x64 (build 19041)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_United States.1252
 ctype    English_United States.1252
 tz       America/Bogota
 date     2022-01-04
 rstudio  2021.09.1+372 Ghost Orchid (desktop)
 pandoc   2.14.0.3 @ C:/Program Files/RStudio/bin/pandoc/ (via rmarkdown)

future::futuresessioninfo() outputs:

*** Package versions
future 1.23.0, parallelly 1.30.0, parallel 4.1.2, globals 0.14.0, listenv 0.8.0

*** Allocations
availableCores():
system 
    12 
availableWorkers():
$system
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[10] "localhost" "localhost" "localhost"

*** Settings
- future.plan=<no set>
- future.fork.multithreading.enable=<no set>
- future.globals.maxSize=<no set>
- future.globals.onReference=<no set>
- future.resolve.recursive=<no set>
- future.rng.onMisuse=<no set>
- future.wait.timeout=<no set>
- future.wait.interval=<no set>
- future.wait.alpha=<no set>
- future.startup.script=<no set>

*** Backends
Number of workers: 2List of future strategies:
1. multisession:
   - args: function (..., envir = parent.frame(), workers = 2)
   - tweaked: TRUE
   - call: plan(multisession, workers = 2)
*** Basic tests
Error in makeClusterPSOCK(workers, ...) : 
  Cluster setup failed. %d of %d workers failed to connect.22
In addition: Warning messages:
1: In system(test_cmd, intern = TRUE, input = input) :
  running command '"C:/PROGRA~1/R/R-41~1.2/bin/x64/Rscript" -e "try(suppressWarnings(cat(Sys.getpid(),file=\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=1.parallelly.parent=21780.551467c36865.pid\")), silent = TRUE)" -e "file.exists(\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=1.parallelly.parent=21780.551467c36865.pid\")"' had status 1
2: In system(test_cmd, intern = TRUE, input = input) :
  running command '"C:/PROGRA~1/R/R-41~1.2/bin/x64/Rscript" -e "try(suppressWarnings(cat(Sys.getpid(),file=\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=2.parallelly.parent=21780.551428d01d7a.pid\")), silent = TRUE)" -e "file.exists(\"C:/Users/Felipe/AppData/Local/Temp/RtmpuYzmcY/worker.rank=2.parallelly.parent=21780.551428d01d7a.pid\")"' had status 1

[HenrikBengtsson]

Hi. Can you please edit your post to use a code block for that warning output. After that, what does future::futureSessionInfo() output?

[fnavarro94]

Absolutely, done.

[HenrikBengtsson]

Got it. All packages are up-to-date. For some reason, the internal initial tests (part of parallelly) used to check if you can even launch a background worker fails. Does the following work for you:

rscript <- file.path(R.home("bin"), "Rscript")
rscript
#> [1] "C:/PROGRA~1/R/R-41~1.2/bin/x64/Rscript"
system2(rscript, args = "--version")
#> R scripting front-end version 4.1.2 (2021-11-01)

If that works, what about:

system2(rscript, args = c("-e", "Sys.getpid()"))
#> [1] 2840 # you'll get another number

? And if that works, what about:

pidfile <- normalizePath(tempfile(), winslash = "/", mustWork = FALSE)
#> [1] "C:/Users/hb/AppData/Local/Temp/RtmpITmm6Q/file1eb8cae79a1"
cmd <- sprintf('cat(Sys.getpid(),file="%s")', pidfile)
cmd
#> [1] "cat(Sys.getpid(),file=\"C:/Users/hb/AppData/Local/Temp/RtmpITmm6Q/file1eb8cae79a1\")"
res <- system2(rscript, args = c("-e", shQuote(cmd)))
res
#> [1] 0

?

[fnavarro94]

Ok, so I ran

 system2(rscript, args = c("-e", "Sys.getpid()"))
#> Error in library(RPostgreSQL) : there is no package called 'RPostgreSQL'
#> Execution halted

RPostgreSQL is a library that I load in in my .Rprofile. Anyways, I manually installed the library and now I can run

plan(multisession, workers = 2)

So, great! that is solved. Now I have a different issue. I am trying to run queries in parallel like this:

system.time({
q1 %<-% dbGetQuery(conn, q1_sql)
q2 %<-% dbGetQuery(conn, q2_sql)
q3 %<-% dbGetQuery(conn, q3_sql)

})

but I get the following error

q1
#>Error in (function (classes, fdef, mtable)  : 
#>unable to find an inherited method for function 'dbSendQuery' for signature '"Amazon Athena", "character"'

do you know what is the cause of this? Thanks for the quick responces!

[HenrikBengtsson]

RPostgreSQL is a library that I load in in my .Rprofile. Anyways, I manually installed the library and now ...

I see. That explains it. So, it's always better to load/attach packages during the R startup conditionally on them being installed, e.g.

if (!require("RPostgreSQL")) warning("Tried to attach RPostgreSQL during startup, but it's not available")

This avoids producing an error during the startup process, which you never want. This being just one example. I would even go as far as saying you should not attach packages during the startup if it's just for convenience of not having to type library(...). My $.02

For the dbSendQuery() issue, please post a new discussion topic, because that one might expand to something longer. I'd to keep discussions here short and concise so that they can be useful for future visitors.

[fnavarro94]

Will do. Thanks for the help!

[HenrikBengtsson]

@fnavarro94 found the answer - it was a broken ~/.Rprofile setup;

Ok, so I ran

 system2(rscript, args = c("-e", "Sys.getpid()"))
#> Error in library(RPostgreSQL) : there is no package called 'RPostgreSQL'
#> Execution halted

RPostgreSQL is a library that I load in in my .Rprofile. Anyways, I manually installed the library and now I can run

plan(multisession, workers = 2)

So, great! that is solved. ...

[angelini75]

I had similar problem, but different output. Check it out please

system2(rscript, args = c("-e", "Sys.getpid()"))
R library paths: 
 C:/Users/angel/Documents/R/win-library/4.3
 C:/Program Files/R/R-4.3.1/library
Error in file(filename, "r", encoding = encoding) : 
  cannot open the connection
Calls: source -> file
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
  cannot open file 'packrat/init.R': No such file or directory
Execution halted

In addition:

> future::futureSessionInfo()
*** Package versions
future 1.33.0, parallelly 1.36.0, parallel 4.3.1, globals 0.16.2, listenv 0.9.0

*** Allocations
availableCores():
system 
    16 
availableWorkers():
$system
 [1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
 [9] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"


*** Settings
- future.plan=<not set>
- future.fork.multithreading.enable=<not set>
- future.globals.maxSize=<not set>
- future.globals.onReference=<not set>
- future.resolve.recursive=<not set>
- future.rng.onMisuse=<not set>
- future.wait.timeout=<not set>
- future.wait.interval=<not set>
- future.wait.alpha=<not set>
- future.startup.script=<not set>

*** Backends
Number of workers: 14
List of future strategies:
1. multisession:
   - args: function (..., workers = 14, envir = parent.frame())
   - tweaked: TRUE
   - call: plan(multisession, workers = 14)

*** Basic tests
Main R session details:
    pid     r sysname release     version nodename machine   login    user effective_user
1 31448 4.3.1 Windows  10 x64 build 19044  host001  x86-64 user001 user001        user001
There were 14 warnings (use warnings() to see them)

1:
.
.
.
13:
14: In system(test_cmd, intern = TRUE, input = input) :
  running command '"C:/PROGRA~1/R/R-43~1.1/bin/x64/Rscript" -e "try(suppressWarnings(cat(Sys.getpid(),file=\"C:/Users/angel/AppData/Local/Temp/RtmpS6c5qR/worker.rank=14.parallelly.parent=31448.7ad87ea65e.pid\")), silent = TRUE)" -e "file.exists(\"C:/Users/angel/AppData/Local/Temp/RtmpS6c5qR/worker.rank=14.parallelly.parent=31448.7ad87ea65e.pid\")"' had status 1