file_watching_automation v1.0.0

name: file_watching_automation description: "Watch files and folders and trigger workflows reliably with debounce, partial-write safety, and missed-event recovery; use when a workflow should react to new or changed local files instead of running on a schedule." metadata: { "includeInPrompt": true }

File Watching Automation

Purpose

React to local files arriving or changing: an export lands in a folder, a log rotates, a build drops an artifact, a user saves a document. The hard part is reliability. Files arrive in pieces, editors save several times in a second, watchers miss events while a process sleeps or restarts, and a busy folder can produce hundreds of events for one real change. This skill covers the patterns that make a file-triggered workflow fire once, on a complete file, and recover from anything it missed.

Scope boundaries, one line each:

• api-polling-patterns polls remote APIs for state changes; this skill watches local files and folders.
• scheduled-report-generation fires on a clock; this skill fires on file events, with polling used only as the detection method.
• desktop-ui-automation controls a graphical interface with clicks and keys; this skill never touches the GUI, it reacts to the filesystem.
• repetitive-work-automation frames automation choices in general; this skill covers one trigger type in depth, the filesystem.

Workflow

1. Write the trigger contract first. Name the watched folder, the file patterns that count (extensions, name prefix), the action each settled file triggers, and what done looks like (moved to a processed folder, renamed, logged). A watcher without a contract fires on temp files, its own output, and half-finished copies.

1. Choose the detection method.

• Polling: scan the folder on an interval and compare size and modification time against the last scan. Works everywhere, including network drives and containers where native events are unreliable or absent. Cost grows with folder size and scan frequency. Start with the polling watcher in step 3.
• Native watchers: the operating system pushes events to the process. Linux provides inotify, macOS provides FSEvents, and Windows provides ReadDirectoryChangesW. Libraries wrap these so one program works across platforms. Native events are fast and cheap, but they can be dropped under load and usually stop at the filesystem boundary of the machine they run on.
• Hybrid: native events for speed plus a periodic polling rescan for recovery. This is the right default for anything that must not miss a file.

1. Start from the polling watcher below. Save it as watch.py, then run it.

python
#!/usr/bin/env python3
"""Polling file watcher: fires once per settled file. Stdlib only.

Usage:
    python3 watch.py WATCH_DIR [--interval 0.5] [--stable 2] [--log fired.log]

A file is settled when its size and mtime stay unchanged for --stable
consecutive polls. Partial writes, editor saves, and chunked copies keep
resetting that counter, so the handler runs only on a complete file.
Rewriting a file (new size or mtime) makes it eligible to fire again.
Prints one line per firing: READY <path> <size_bytes>
"""
import argparse
import os
import sys
import time

IGNORE_SUFFIXES = (".tmp", ".part", ".swp", ".swx", "~")


def scan(root):
    """Return {path: (size, mtime_ns)} for regular files under root."""
    found = {}
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in dirnames if not d.startswith(".")]
        for name in filenames:
            if name.startswith(".") or name.endswith(IGNORE_SUFFIXES):
                continue
            path = os.path.join(dirpath, name)
            try:
                st = os.stat(path)
            except OSError:
                continue
            found[path] = (st.st_size, st.st_mtime_ns)
    return found


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("watch_dir")
    ap.add_argument("--interval", type=float, default=0.5,
                    help="seconds between polls (default 0.5)")
    ap.add_argument("--stable", type=int, default=2,
                    help="unchanged polls required before firing (default 2)")
    ap.add_argument("--log", default="",
                    help="optional file to append firing lines to")
    args = ap.parse_args()

    if not os.path.isdir(args.watch_dir):
        print(f"not a directory: {args.watch_dir}", file=sys.stderr)
        return 2

    pending = {}   # path -> [signature, unchanged_poll_count]
    fired = {}     # path -> signature last fired for
    print(f"watching {args.watch_dir}", flush=True)
    try:
        while True:
            current = scan(args.watch_dir)
            # Drop state for files that disappeared.
            for path in list(pending):
                if path not in current:
                    del pending[path]
            for path in list(fired):
                if path not in current:
                    del fired[path]
            for path, sig in current.items():
                if fired.get(path) == sig:
                    continue  # already fired for this exact version
                if path in pending and pending[path][0] == sig:
                    pending[path][1] += 1
                else:
                    pending[path] = [sig, 0]  # new or changed: restart debounce
                if pending[path][1] >= args.stable:
                    line = f"READY {path} {sig[0]}"
                    print(line, flush=True)
                    if args.log:
                        with open(args.log, "a", encoding="utf-8") as fh:
                            fh.write(line + "\n")
                    fired[path] = sig
                    del pending[path]
            time.sleep(args.interval)
    except KeyboardInterrupt:
        return 0


if __name__ == "__main__":
    sys.exit(main())

Run it:

python3 watch.py ./inbox --interval 0.5 --stable 2 --log fired.log

Replace the READY print with the real action (call a function, append to a queue, run a command) once the firing behavior checks out. Keep the firing log either way; it is the audit trail when someone asks whether a file was processed.

1. Enforce partial-write safety. A create event means a file exists. It says nothing about whether the writer has finished. Three defenses, strongest first:

• Atomic rename pattern: producers write to a temp name (data.csv.tmp) and rename to the final name when done. A rename within the same filesystem is atomic, so any file visible under its final name is complete. If you control the producers, require this pattern and watch only final names.
• Size and mtime stability: fire only after the size and modification time stay unchanged across several polls spanning at least a second or two. This is what watch.py does. It handles producers you cannot change, such as FTP drops and application exports.
• Close events: native watchers can report a file being closed after writing. Treat a close as a hint, then still apply the stability check, because some writers close and reopen the same file repeatedly.

1. Debounce bursts. Editors, sync tools, and build systems often save the same file many times in a second. Every new size or mtime resets the stability counter in watch.py, so a burst collapses into one firing after the last save. For native watchers, apply the same rule in the handler: record the latest event per path, wait for a quiet window, then act once.

1. Add missed-event recovery. Every watcher misses events: the process was down, the machine slept, the event queue overflowed, the network drive dropped. Because watch.py rescans the whole folder on every poll, a file that appeared while it was down still fires on the next run, as long as its signature was never recorded as fired. Pair native watchers with the same periodic full rescan (every few minutes is typical) and reconcile: any file present on disk with no fired record gets processed. Persist the fired state to disk if a restart must not reprocess old files; the in-memory version in watch.py refires everything settled after a restart, which is safe only when the handler is idempotent.

1. Handle event storms. A bulk copy, an unzip, or a build can drop thousands of files at once. Protect the workflow:

• Coalesce by path before acting, so repeated events for one file become one job.
• Queue jobs and process them at a fixed rate instead of spawning a handler per event.
• Cap concurrent handlers and set a maximum queue size; when the cap is hit, log the overflow and let the next rescan pick the files up.
• Watch out for feedback loops: if the handler writes into the watched folder, exclude its output paths, or the watcher will trigger itself forever.

1. Respect recursive watch limits. Native watchers register each subdirectory separately, and operating systems cap how many watches one process or user can hold. The cap is system-configured and varies; on the Linux machine used to test this skill, /proc/sys/fs/inotify/max_user_watches read 63753, and other machines will differ. Deep or wide trees can hit the cap and then fail silently or with errors at registration time. Polling avoids per-directory watches entirely, which is one reason it suits very large trees. Whatever the method, count the directories before deploying and test at real tree size.

1. Consider a native library only when polling is too slow or too coarse. The Python library watchdog (version 6.0.0 on PyPI at drafting time, requires Python 3.9 or later) provides an Observer and a FileSystemEventHandler with an on_any_event method, per the example in its PyPI project description:

python
import time

from watchdog.events import FileSystemEvent, FileSystemEventHandler
from watchdog.observers import Observer


class Handler(FileSystemEventHandler):
    def on_any_event(self, event: FileSystemEvent) -> None:
        print(event)


observer = Observer()
observer.schedule(Handler(), ".", recursive=True)
observer.start()
try:
    while True:
        time.sleep(1)
finally:
    observer.stop()
    observer.join()

Install it with pip install watchdog and confirm the installed version matches what the code was tested against. Keep the stability check, debounce, and rescan from steps 4 to 6 in place; a native event stream needs them just as much as polling does.

1. Test before trusting it. Run the watcher against a scratch folder and confirm each case:

• A file written in chunks with pauses fires exactly once, after the final chunk.
• A file saved twice in quick succession fires once.
• A file rewritten after processing fires again, as a new version.
• A temp-suffix file (.tmp, .part) never fires.
• A file added while the watcher is stopped fires after restart (polling and rescan designs), or the miss is documented as accepted behavior.
• A burst of many files fires once per file, at a controlled rate.

Output Contract

A file watching setup delivered with this skill includes:

• The trigger contract: watched folder, matching patterns, ignored patterns, and the action per settled file.
• The detection choice (polling, native, or hybrid) with the reason, stated in one or two sentences.
• The runnable watcher, self-contained, with the debounce interval, stability requirement, and rescan interval set explicitly.
• The partial-write defense in use: atomic rename required from producers, stability check, or both.
• A firing log format that records path, size, and time for every firing, so processing can be audited and duplicates can be proven or ruled out.
• Test results for the six cases in step 10, with the actual firing counts observed.

Operating Rules

1. Never act on a create event alone. Act on a settled file: stable size and mtime, a completed atomic rename, or a close event confirmed by a stability check.
2. Make the handler idempotent. The same file version can reach it twice through a restart or a rescan; the second run must be harmless.
3. One firing per file version. A version is the path plus its size and mtime signature. A changed signature is a new version and may fire again.
4. Keep watcher output out of the watched set. Exclude the firing log, processed folders, and any files the handler writes, by path or suffix.
5. Assume events will be missed and design the rescan to catch them. A watcher with no reconciliation path will lose files quietly.
6. Bound the work: fixed poll or rescan interval, coalescing per path, a capped queue, and a capped number of concurrent handlers.
7. Check recursive limits at real scale before deploying on a large tree. Watch-count caps are set by the operating system and differ between machines.
8. Log every firing and every overflow. When a file goes missing from a workflow, the log is how you tell a watcher miss apart from a handler failure.
9. Prefer the standard library polling watcher for new setups. Add a native library when measurement shows polling latency or scan cost is a real problem, and pin the version you tested.