Python Script

This guide demonstrates the basic concepts around creating and executing a Python script service within torero.

Prerequisites

It is a prerequisite that a Git repository be setup with a Python script within it. Review the Create Repository command to understand on how to create a repository.

Create

torero create service python-script will perform a creation of the Python script service. A more detailed guide of all creation options is available there but this guide should help get you started.

The command shown below creates a Python script service within the torero called simple-python. We are leveraging a repository that would have been previously configured called torero-resources.

>_ torero create service python-script simple-python --repository torero-resources --working-dir pythonscripts --filename main.py

It is important to stop and understand the structure of torero-resources before moving further.

torero-resources (git repo)

├── README.md
├── ansibleplaybooks
├── pythonscripts
│   ├── main.py
│   └── requirements.txt
└── opentofuplans

We specified that we want to use torero-resources via the --repository flag.

Notice that our Python script exists in a directory called pythonscripts. We denoted this using the --working-dir flag.

Inside the pythonscripts directory, we have an actual script called main.py. We denoted this with the --filename flag.

If your Python script requires external libraries, you can place a requirements.txt file within the working directory. torero will automatically manage the dependencies for you. If you have a Python project that uses pyproject.toml, please reference the dedicated section later on in this document.

Verify Creation

We can view details about the previously created Python script service by running the describe service command.

>_ torero describe service simple-python
Output:

Name:        simple-python
Repo Name:   torero-resources
Working Dir: pythonscripts
FileName:    main.py
Decorator:
Env Vars:
Description:
Tags:

Execution

Executing a Python script is simple from torero by utilizing the run service python-script command.

Consider the contents of our main.py Python script shown below

# main.py
from netmiko import ConnectHandler
import argparse

def main():
    parser = argparse.ArgumentParser(description="Netmiko Command Service")
    parser.add_argument('--command', required=True, help="Command to run")
    parser.add_argument('--host', required=True, help="Host to connect to")
    args = parser.parse_args()
    # YOUR EXECUTION CODE GOES BELOW THIS LINE    

    cisco1 = {
        "device_type": "cisco_ios",
        "host": args.host
    }
    with ConnectHandler(**cisco1) as net_connect:
        output = net_connect.send_command(args.command)

    print(output)

if __name__ == "__main__":
    main()

Our script requires the netmiko library. This is specified in the requirements.txt file located in the services working-dir. Let's look at the contents of our requirements.txt file below.

netmiko==4.3.0

When executing a Python script using the run command, torero will automatically look for a requirements.txt file in the working directory that was specified during the service's creation. torero will then create the virtual environment with all the dependencies specified in the requirements.txt and execute the script within that virtual environment. This is a very powerful feature that allows for easy management of Python dependencies.

We can observe that oru script takes in two inputs: command abd host. They can be specified using the --set flag.

>_ torero run service python-script simple-python --set host=10.0.0.1 --set command="show version"
Output:

Start Time:   2024-01-01T12:00:00Z
End Time:     2024-01-01T12:00:01Z
Elapsed Time: 1.372672s
Return Code: 0
================================================================================
                           System Version Information
================================================================================

...truncated for document

Stderr:

In the backend, torero will take they key=values defined via the --set command on the run command and pass it to the Python service. In our example, the following command would be run by torero in the backend: <python-executable> main.py --host=10.0.0.1 --command="show version".

Barebones Script With Inputs

The most important part of working with a Python script is how to get the arguments into the script. The contract between torero and a Python service is the ability to read key=value from the command line. Every Python script that is written will have something that looks like the following Python code. You can follow the barebones example to get started with accepting input arguments.

# main.py
import argparse

def main():
    parser = argparse.ArgumentParser(description="torero example")
    parser.add_argument('--name', required=True, help="Name to greet")
    args = parser.parse_args()
    name = args.name
    # YOUR EXECUTION CODE GOES BELOW THIS LINE    

    print(f"Hello, {name}")

if __name__ == "__main__":
    main()

Pyproject.toml

torero can utilize pyproject.toml files to manage dependencies and build projects. Consider the following repository layout.

pyproject-repo (git repo)

├── README.md
├── pyproj-example
│   ├── pyproject.toml
│   └── src
│       ├── __init__.py
└──     └── __main__.py

And the following pyproject.toml file

[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"

[project]
name = "pyproj-example"
version = "1.0.0"
dependencies = [
  "requests",
]

[project.optional-dependencies]
networking-deps = ["netmiko"]

[project.scripts]
some-script = "src.__main__:main"

[tool.setuptools]
packages = ["src"]

torero's create service python-script command supports the following flags to specify how it should handle your pyproject.toml file.

  • --req-file - The path to the requirements file. This could also be a path deeper than the --working-dir
  • --pyproj-script - Defines the script to execute as would be defined in the pyproject.toml
  • --pyproj-optional-deps - Specifies optional dependencies defined in the pyproject.toml to install before executing

Given the example above, a user can execute the following command to create a service that will use pyproject.toml to: install the required dependencies, the optional dependencies of networking-deps, and build/execute the some-script script at runtime.

>_ torero create service python-script simple-toml --repository pyproject-repo --working-dir pyproj-example --pyproj-script some-script  --req-file pyproject.toml --pyproj-optional-deps networking-deps

When this script is executed, a virtual environment will be built with both the regular dependencies and optional networking-deps dependencies. The some-script script will also be built using the build system defined in the pyproject.toml. In our case, it is setup-tools. Finally, torero will execute the script on the CLI with syntax that is similar to what is shown below.

/path/to/venv/bin/some-script

Specifying The Default Requirements File

By default, if no requirements file is specified via --req-file, torero will look for the requirements file specified by the configuration variable TORERO_PYTHON_SCRIPT_REQUIREMENTS_FILE. The default value for TORERO_PYTHON_SCRIPT_REQUIREMENTS_FILE is requirements.txt but it can also be set to any other filepath that ends in requirements.txt or pyproject.toml. The filepath will always be relative to the working-dir of the service.

Behind The Scenes

Below are flowcharts depicting how torero manages virtual environments and dependencies when using either requirements.txt and pyproject.toml.

You do not need to understand this to use torero, but it can be a useful reference.

Requirements.txt

flowchart TD A[Start Python Service] --> B{"Does requirements.txt exist?"} B -- No --> C[Generate hash from empty string] B -- Yes --> D[Get contents of requirements.txt] D --> E[Generate hash from the contents] C --> F E --> F{"Does a venv exist with this hash?"} F -- No --> G[Find Python Interpreter] G --> H[Create Virtualenv] H --> I[Install Requirements] F -- Yes --> J[Use existing venv] J --> K I--> K[Run Python script with venv]

Pyproject.toml

flowchart TD A[Start Python Service] --> B{"Does pyproject.toml exist?"} B -- No --> C[Error out] B -- Yes --> D[Get SHA of current commit in repository] D --> E[Get working dir of service] E --> F[Get path to pyproject.toml] F --> G[Get optional dependency names] G --> H[Combine all of these fields and generate a hash off of them] H --> I{"Does a venv exist with this hash?"} I -- No --> J[Find Python Interpreter] J --> K[Create Virtualenv] K --> L["Build project within venv using pyproject contents"] I -- Yes --> M[Use existing venv] L --> O M --> O[Run Python script with venv]

Decorators

It is possible to put restrictions around the inputs that are accepted by a Python script by utilizing decorators. For more information on decorators in general, please refer to this guide.

CLI Reference

Python Script Create

Python Script Run

Services Get

Service Describe

Service Delete