The Challenge: Live Console Streaming vs. Output Capture in R

When running long-running external system commands using R's built-in system() or system2() functions, developers often face a common trade-off: you can either stream the command output live to the R console or redirect it to capture it in a variable or file. Attempting to pass stdout = TRUE or a file path suppresses the live console updates, leaving you in the dark until the process completes.

If you are executing a long process and need both immediate visual feedback in the console and the ability to log or process the output afterward, standard base R functions fall short. Fortunately, there is a powerful, modern, and cross-platform solution available in R using the processx package.

The Modern Solution: Using the processx Package

The processx package is designed to execute external processes asynchronously or synchronously with fine-grained control over standard output and error streams. By using its stdout_line_callback feature, you can stream lines directly to the console as they are generated while still storing the full output in the returned object.

Step-by-Step Example

First, install and load the package if you haven't already:

install.packages("processx")
library(processx)

Here is how to run an external Rscript process, display its progress in real-time, and save the result:

# Command arguments to simulate a long-running process
args <- c("-e", "for(i in 1:5) { Sys.sleep(1); print(paste('Step', i, 'at', Sys.time())) }")

# Execute process with real-time output and capture
result <- processx::run(
  command = "Rscript",
  args = args,
  stdout_line_callback = function(line, proc) {
    cat(line, "\n") # Print live line to console
  },
  stderr_line_callback = function(line, proc) {
    message(line)   # Print stderr messages if needed
  }
)

# Access the complete captured stdout
captured_text <- result$stdout
cat("\n--- Output stored in variable ---\n")
cat(captured_text)

Writing Live Output to a Log File Simultaneously

If your goal is to display progress on the screen while continuously appending output to an external log file, you can modify the callback function:

log_file <- "process_log.txt"

result <- processx::run(
  command = "Rscript",
  args = args,
  stdout_line_callback = function(line, proc) {
    # Print live output to R console
    cat(line, "\n")
    
    # Append live output to external file
    cat(line, "\n", file = log_file, append = TRUE)
  }
)

Why Choose processx over Base R?

  • Cross-Platform Consistency: Works reliably across Windows, macOS, and Linux without platform-specific shell tricks like Unix tee or PowerShell redirections.
  • Non-Blocking Control: Handles line buffering efficiently so that output appears in real-time rather than in delayed chunks.
  • Separate Error Handling: Stream or capture stdout and stderr independently without polluting standard logs.

Conclusion

While base R functions like system2() force a choice between capturing output and printing progress, processx::run() provides a clean, cross-platform solution to achieve both simultaneously. Using callback streams ensures your users stay informed with live feedback while your script safely stores logs for further processing.