# HG changeset patch # User siwaa # Date 1695659151 0 # Node ID 2a6d7d053ba75b8b4f6fb14a3e6496da07d0d9c5 # Parent 0625e6a639eaf421e69d5641900b52c692af6a41 "planemo upload for repository https://forgemia.inra.fr/redelac commit adcac73e150f1f821b4b7f7673a490dc0fadc936" diff -r 0625e6a639ea -r 2a6d7d053ba7 redelacSticsInputGenerator.R --- a/redelacSticsInputGenerator.R Wed Aug 30 08:23:12 2023 +0000 +++ b/redelacSticsInputGenerator.R Mon Sep 25 16:25:51 2023 +0000 @@ -1,7 +1,10 @@ library(SticsRFiles) library(dplyr) library(lubridate) +library(parallel) +library(doParallel) +sessionInfo() startTime <- Sys.time() @@ -19,6 +22,8 @@ args = commandArgs(trailingOnly = TRUE) genUSMsXmlOneAfterOther <- (args[1] == "usmx_one_after_other") +parallel <- (args[2] == "parallel") +nbSlots <- as.integer(args[3]) # USMs file loading @@ -68,97 +73,120 @@ gen_sta_xml(param_df = sta_param, out_dir = workspace) if (genUSMsXmlOneAfterOther) { + durGenX <- lubridate::as.duration(0) durGenT <- lubridate::as.duration(0) - + + if (parallel) { + cl <- makeCluster(nbSlots) + print(cl) + registerDoParallel(cl) + `%dordopar%` <- `%dopar%` + } else { + `%dordopar%` <- `%do%` + } + # looping on USMs - for (i in 1:nrow(usms_param)) { - row <- usms_param[i, ] - + loopRes <- foreach::foreach(i = 1:nrow(usms_param), + .packages = c("SticsRFiles", "lubridate")) %dordopar% { + beforeGenXmlTimeI <- Sys.time() - - gen_usms_xml(file = file.path(workspace, "usms.xml"), - param_df = row) - + + usmxFileName = paste0("usms", as.character(i) , ".xml") + + usmxFilePath = file.path(workspace, usmxFileName) + + gen_usms_xml(file = usmxFilePath, + param_df = usms_param[i, ]) + afterGenXmlTimeI <- Sys.time() durGenXI <- lubridate::as.duration(lubridate::interval(beforeGenXmlTimeI, afterGenXmlTimeI)) - + # one after the other generating txt folder - + beforeGenTxtTimeI <- Sys.time() - + gen_usms_xml2txt( + usms_file = usmxFileName, javastics = javastics_path, workspace = workspace, out_dir = txt_path, verbose = TRUE ) - + afterGenTxtTimeI <- Sys.time() durGenTI <- lubridate::as.duration(lubridate::interval(beforeGenTxtTimeI, afterGenTxtTimeI)) - - durGenX <- durGenX + durGenXI - durGenT <- durGenT + durGenTI + + return(c(durGenXI, durGenTI)) } + + if (parallel) stopCluster(cl) + + durDS <- as.data.frame(do.call(rbind, loopRes)) + durGenX <- sum(durDS[1]) + durGenT <- sum(durDS[2]) + } else { + beforeGenXmlTime <- Sys.time() - + gen_usms_xml(file = file.path(workspace, "usms.xml"), param_df = usms_param) - + afterGenXmlTime <- Sys.time() durGenX <- lubridate::as.duration(lubridate::interval(beforeGenXmlTime, afterGenXmlTime)) - + beforeGenTxtTime <- Sys.time() - + gen_usms_xml2txt( javastics = javastics_path, workspace = workspace, out_dir = txt_path, verbose = TRUE ) - + afterGenTxtTime <- Sys.time() durGenT <- lubridate::as.duration(lubridate::interval(beforeGenTxtTime, afterGenTxtTime)) - + } endTime <- Sys.time() + dur <- lubridate::as.period(lubridate::as.duration(lubridate::interval(startTime, endTime))) durGenXP <- lubridate::as.period(durGenX) durGenTP <- lubridate::as.period(durGenT) -print("REDELAC Report") -print("==============") +print("REDELACSticsInputGenerator Report") +print("=================================") print(paste0("number of usms: ", nrow(usms_param))) print(paste("usms.xml one after the other:", genUSMsXmlOneAfterOther)) + sprintf( 'overall duration: %g jour(s) %g heure(s) %g minute(s) %g seconde(s)', lubridate::day(dur), lubridate::hour(dur), lubridate::minute(dur), - lubridate::second(dur) -) + lubridate::second(dur)) + sprintf( 'inside gen_usms_xml duration: %g jour(s) %g heure(s) %g minute(s) %g seconde(s)', lubridate::day(durGenXP), lubridate::hour(durGenXP), lubridate::minute(durGenXP), - lubridate::second(durGenXP) -) + lubridate::second(durGenXP)) + sprintf( 'inside gen_usms_xml2txt duration: %g jour(s) %g heure(s) %g minute(s) %g seconde(s)', lubridate::day(durGenTP), lubridate::hour(durGenTP), lubridate::minute(durGenTP), - lubridate::second(durGenTP) -) + lubridate::second(durGenTP)) diff -r 0625e6a639ea -r 2a6d7d053ba7 redelacSticsInputGenerator.bash --- a/redelacSticsInputGenerator.bash Wed Aug 30 08:23:12 2023 +0000 +++ b/redelacSticsInputGenerator.bash Mon Sep 25 16:25:51 2023 +0000 @@ -1,20 +1,20 @@ -cp -r /home/rstudio/JavaSTICS-1.5.1-STICS-10.0.0/JavaSTICS-1.5.1-STICS-10.0.0 . && +cp -r /home/rstudio/JavaSTICS-1.5.1-STICS-10.0.0 . && rm -rf JavaSTICS-1.5.1-STICS-10.0.0/config && -cp -r $2/config JavaSTICS-1.5.1-STICS-10.0.0 && -unzip -q $3 -d TMP && +cp -r $4/config JavaSTICS-1.5.1-STICS-10.0.0 && +unzip -q $5 -d TMP && export WSname=$(ls TMP) && mv TMP/* JavaSTICS-1.5.1-STICS-10.0.0/WS && cp JavaSTICS-1.5.1-STICS-10.0.0/config/param_gen.xml JavaSTICS-1.5.1-STICS-10.0.0/WS && cp JavaSTICS-1.5.1-STICS-10.0.0/config/param_newform.xml JavaSTICS-1.5.1-STICS-10.0.0/WS && -unzip -q $4 -d TMPbis && +unzip -q $6 -d TMPbis && mv TMPbis/*/* JavaSTICS-1.5.1-STICS-10.0.0/WS && -cp $5 JavaSTICS-1.5.1-STICS-10.0.0/WS/USMs.csv && -cp $6 JavaSTICS-1.5.1-STICS-10.0.0/WS/Tec.csv && -cp $7 JavaSTICS-1.5.1-STICS-10.0.0/WS/Ini.csv && -cp $8 JavaSTICS-1.5.1-STICS-10.0.0/WS/Station.csv && +cp $7 JavaSTICS-1.5.1-STICS-10.0.0/WS/USMs.csv && +cp $8 JavaSTICS-1.5.1-STICS-10.0.0/WS/Tec.csv && +cp $9 JavaSTICS-1.5.1-STICS-10.0.0/WS/Ini.csv && +cp ${10} JavaSTICS-1.5.1-STICS-10.0.0/WS/Station.csv && cd JavaSTICS-1.5.1-STICS-10.0.0 && -Rscript --verbose $2/redelacSticsInputGenerator.R $1 && +Rscript --verbose $4/redelacSticsInputGenerator.R $1 $2 $3 && rm WS/1* && mv WS $WSname && zip -q -r $WSname $WSname && -cp $WSname.zip $9 +cp $WSname.zip ${11} diff -r 0625e6a639ea -r 2a6d7d053ba7 redelacSticsInputGenerator.xml --- a/redelacSticsInputGenerator.xml Wed Aug 30 08:23:12 2023 +0000 +++ b/redelacSticsInputGenerator.xml Mon Sep 25 16:25:51 2023 +0000 @@ -1,12 +1,17 @@ - + to run generate Stics Inputs for REDELAC - docker://registry.forgemia.inra.fr/redelac/redelac-toolbox/sticsrfiles:9a2d54c0674a6b902299dd22fdd37c2f67509fa2 + docker://registry.forgemia.inra.fr/redelac/redelac-toolbox/sticsrpacks:b782e0b7ce3af7e7a14a87ea750145ed1c2595a0 { + #if str( $IterUsmX.IterUsmX ) == 'usmx_one_after_other': + parallel=${$IterUsmX.parallel} && + #else + parallel=parallel && + #end if + bash $__tool_directory__/redelacSticsInputGenerator.bash ${IterUsmX.IterUsmX} \${parallel} \${GALAXY_SLOTS:-4} $__tool_directory__ ${Workspace} ${Climat} ${USMs} ${Tec} ${Ini} ${Station} ${GeneratedWorkspace} + ]]> @@ -16,7 +21,14 @@ - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff -r 0625e6a639ea -r 2a6d7d053ba7 test-data/input/.~lock.USMsSheet.csv# --- a/test-data/input/.~lock.USMsSheet.csv# Wed Aug 30 08:23:12 2023 +0000 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,1 +0,0 @@ -,pchabrier,llosas.toulouse.inra.fr,28.08.2023 13:06,file:///home/pchabrier/.config/libreoffice/4; \ No newline at end of file