Python subprocess

The subprocess module in Python provides tools to start and manage additional processes from within a Python script, interact with them, capture output, and more.

1. Importing the subprocess Module

First, import the subprocess module to use its functionalities.

import subprocess

2. Running a Simple Command Using subprocess.run()

The subprocess.run() function is the recommended way to run commands. It blocks execution until the command completes and returns a CompletedProcess object containing details about the executed command.

# Run the 'echo' command
result = subprocess.run(["echo", "Hello, World!"], capture_output=True, text=True)
print("Return Code:", result.returncode)
print("Output:", result.stdout)

Output:

Return Code: 0
Output: Hello, World!

Explanation: The capture_output=True argument captures the output, and text=True returns the output as a string.

3. Running a Command with Shell Integration

By setting shell=True, you can run commands as if in a shell. This allows usage of shell-specific features, like pipes or wildcard expansion.

# Run a shell command with wildcard
result = subprocess.run("echo Hello, World!", shell=True, capture_output=True, text=True)
print("Output:", result.stdout)

Output:

Output: Hello, World!

Explanation: shell=True allows the command to run in the shell environment, where wildcards and pipes work.

4. Capturing Both stdout and stderr

Using capture_output=True, both standard output and standard error can be captured.

# Attempt to run a command that fails
result = subprocess.run(["ls", "nonexistent_file"], capture_output=True, text=True)
print("Return Code:", result.returncode)
print("Output:", result.stdout)
print("Error:", result.stderr)

Output:

Return Code: 2
Output: 
Error: ls: cannot access 'nonexistent_file': No such file or directory

Explanation: stderr contains error messages, allowing error handling and debugging.

5. Using subprocess.Popen for Non-blocking Execution

Popen is used for more control over process execution, allowing commands to run asynchronously.

# Run a command asynchronously
process = subprocess.Popen(["sleep", "2"])
print("Process started. Waiting for it to finish...")
process.wait()
print("Process finished.")

Output:

Process started. Waiting for it to finish...
Process finished.

Explanation: wait() waits for the process to complete before continuing the script.

6. Communicating with a Process Using Popen

Use communicate() to send input and capture output/error when interacting with the process.

# Start a process and communicate with it
process = subprocess.Popen(["grep", "Hello"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
output, error = process.communicate("Hello, subprocess!")
print("Output:", output)

Output:

Output: Hello, subprocess!

Explanation: communicate() sends data to the process’s standard input and captures the output. Here, it filters the input text for the word "Hello".

7. Using check_output for Output Only

The check_output() function runs a command and returns only the standard output. It raises an exception if the command fails.

try:
    output = subprocess.check_output(["echo", "Hello, check_output!"], text=True)
    print("Output:", output)
except subprocess.CalledProcessError as e:
    print("An error occurred:", e)

Output:

Output: Hello, check_output!

Explanation: check_output() is ideal for commands that should succeed, returning only the output without detailed process information.

8. Handling Exceptions in subprocess

If a command fails, subprocess.CalledProcessError is raised. This is useful for error handling in scripts.

try:
    # Attempt to list a non-existent directory
    subprocess.check_output(["ls", "nonexistent_dir"], stderr=subprocess.STDOUT, text=True)
except subprocess.CalledProcessError as e:
    print("Error occurred:")
    print("Return Code:", e.returncode)
    print("Output:", e.output)

Output:

Error occurred:
Return Code: 2
Output: ls: cannot access 'nonexistent_dir': No such file or directory

Explanation: Capturing both return code and output provides helpful error context.

9. Redirecting stdout and stderr to a File

Redirecting output and errors to a file can be done by opening a file and passing it to subprocess.run().

# Open a file for writing
with open("output.log", "w") as file:
    # Run command and redirect output and error to file
    subprocess.run(["echo", "Redirecting to a file"], stdout=file, stderr=file)
print("Output redirected to output.log.")

Output:

Output redirected to output.log.

Explanation: By redirecting to output.log, no output appears in the terminal, making it useful for logging.

10. Summary

The subprocess module offers comprehensive process management and control, allowing Python scripts to run and interact with system commands. From simple commands to complex pipelines and error handling, subprocess provides robust tools for integrating external processes.