sshdol

A Python dict-like interface to SSH files, built on dol (Data Object Layer).

What is sshdol?

sshdol provides a familiar mapping (dict-like) interface to interact with files on remote servers via SSH. Instead of dealing with low-level SSH operations, you can interact with remote files just like you would with Python dictionaries:

s['file.txt'] = b'content'   # Write to a file
content = s['file.txt']      # Read from a file
'file.txt' in s              # Check if file exists
list(s)                      # List all files and directories

Why use sshdol?

dol helps create key-value facades to data sources and systems that have Python builtin interfaces. This offers a consistent, easy, and familiar interface to various systems, making it healthier for your code and your developer sanity.

Installation

pip install sshdol

Quick Start

To use sshdol, you need access to an SSH server and appropriate credentials.

from sshdol.base import SshFiles, SshTextFiles

# Connect to a server (several authentication methods supported)
s = SshFiles(host="myserver", rootdir="/path/to/safe/directory")

# Write binary data to a file
s['example.bin'] = b'binary content'

# Read the file
content = s['example.bin']  # Returns b'binary content'

# List files and directories
files = list(s)  # Returns [..., 'example.bin', ...]

# Check if a file exists
if 'example.bin' in s:
    print("File exists!")

Features

Fast sync (to local folder)

There are two ways to copy/sync a remote SSH directory into a local folder.

• Mapping-based (simple but slow on many files):

import dol
from sshdol.base import SshFiles

src = SshFiles(host="myserver", rootdir="/remote/path")
target = dol.Files("/local/target/folder")

# This works and preserves dol’s familiar Mapping API, but can be very slow
target.update(src)

You can also use higher-level helpers such as xdol.update_with_policy to control sync behavior (e.g., mirror vs copy), but the round-trips remain expensive for large trees. (Note: Both of these are applicable to any Mapping instance src and MutableMapping instance target.)

• High-performance (recommended for many files):

Use SshFiles.sync_to, which leverages rsync under the hood for orders-of-magnitude faster syncs. It only supports local folder targets (or objects that expose a filesystem root via a rootdir attribute, e.g., dol.Files).

from sshdol.base import SshFiles
import dol

src = SshFiles(host="myserver", rootdir="/remote/path")

# Sync to a local path (fast; uses rsync)
src.sync_to("/local/target/folder")

# Or sync to a dol.Files instance (it has a .rootdir pointing to a local folder)
dst = dol.Files("/local/target/folder")
src.sync_to(dst)

Notes:

• Requires rsync installed locally.
• Target must be a local filesystem path or an object with a .rootdir that is one (e.g., dol.Files).
• You can control deletion timing and safety:

# Remove local files not present on remote (mirror behavior)
src.sync_to(
    "/local/target/folder",
    delete_local_files_not_in_remote=True,
    delete_mode="during",  # 'before' | 'after' | 'delay' | 'during'
)

# “Recycle” deletions instead of permanently deleting them
src.sync_to(
    "/local/target/folder",
    delete_mode="recycle",            # uses rsync --backup/--backup-dir
    recycle_bin="~/.Trash",           # optional (defaults OS-specifically)
)

Compared to target.update(src), sync_to performs a single rsync negotiation and transfers only changed file blocks, making it dramatically faster on large trees.

Binary and Text Mode

sshdol provides both binary and text interfaces:

# Binary mode (default)
s = SshFiles(host="myserver", rootdir="/path/to/directory")
s['file.txt'] = b'binary content'

# Text mode
t = SshTextFiles(host="myserver", rootdir="/path/to/directory")
t['file.txt'] = 'text content'  # Automatically handles encoding/decoding

Recursive Directory Support

Navigate through directories and access nested files with slash-separated paths:

# Create a store with unlimited depth and directory creation
s = SshFiles(
    host="myserver",
    rootdir="/path/to/directory",
    max_levels=None,  # Unlimited recursion for listing
    create_dirs=True  # Create directories as needed
)

# Write to a deeply nested path (creates directories if they don't exist)
s['dir1/dir2/dir3/file.txt'] = b'nested content'

# Read from the nested path
content = s['dir1/dir2/dir3/file.txt']  # Returns b'nested content'

# List will show all files and directories recursively
all_files = list(s)  # [..., 'dir1/', 'dir1/dir2/', 'dir1/dir2/dir3/', 'dir1/dir2/dir3/file.txt', ...]

Directory Navigation

Navigate through directories by accessing folders like dictionary keys:

# Get a store instance for a subdirectory
subdir = s['dir1/']

# List only files in the subdirectory
subdir_files = list(subdir)  # [..., 'dir2/', ...]

# Navigate further
subsubdir = subdir['dir2/']

Recursion Control

Control the depth of directory recursion:

# Only show top-level files and directories
s0 = SshFiles(host="myserver", max_levels=0)
files_level0 = list(s0)  # [..., 'dir1/', 'file.txt', ...]

# Show files up to one level deep
s1 = SshFiles(host="myserver", max_levels=1)
files_level1 = list(s1)  # [..., 'dir1/', 'dir1/file.txt', 'dir1/dir2/', ...]

Directory Creation

Automatically create directories when writing to nested paths:

# Create directories as needed when writing
s = SshFiles(host="myserver", create_dirs=True)

# This will create all necessary directories
s['new/path/to/file.txt'] = b'content with auto-created directories'

Authentication Options

sshdol supports multiple SSH authentication methods:

# Using SSH config alias (simplest if you have ~/.ssh/config set up)
s = SshFiles(host="myserver")

# Using username/password
s = SshFiles(user="username", password="password", url="example.com")

# Using SSH key
s = SshFiles(user="username", url="example.com", key_filename="~/.ssh/id_rsa")

Advanced Examples

Working with text files

# Create a text file store
t = SshTextFiles(host="myserver")

# Write a text file
t['notes.txt'] = 'Hello, this is a text note'

# Read it back
text = t['notes.txt']  # Returns 'Hello, this is a text note'

Creating a new directory

Now, know that your ssh stores have a mkdir where you can make a directory, which can then be written in:

# Create a new directory
projects_dir = s.mkdir('projects')

# Write files in the new directory
projects_dir['README.md'] = b'# Projects Directory'

But if you specify create_dirs=True, you are allowing the store to make these directories automatically, at the moment you want to write in them (sort of like what github and some blob storage systems do):

# Create a store with directory creation enabled
s = SshFiles(host="myserver", create_dirs=True)

s['some_folder/that/did/not/exist/README.md'] = b'Write me!'

Handling large directory structures

# Create a store with limited recursion for better performance
s = SshFiles(host="myserver", max_levels=2)

# List files up to 2 levels deep
files = list(s)  # [..., 'dir1/', 'dir1/dir2/', 'dir1/dir2/file.txt', ...]

# You can still access deeper files directly
deep_content = s['dir1/dir2/dir3/dir4/file.txt']

License

MIT