Osh

Osh (Object SHell) is a tool that integrates the processing of structured data, database access, and remote access to a cluster of nodes. These capabilities are made available through a command-line interface (CLI) and a Python application programming interface (API).

Osh processes streams of Python objects using simple commands. Complex data processing is achieved by command sequences in which the output from one command is passed to the input of the next. This is similar to composing Unix commands using pipes. However, Unix commands pass strings from one command to the next, and the commands (grep, awk, sed, etc.) are heavily string-oriented. Osh commands send primitive Python types such as strings and numbers; composite types such as tuples, lists and maps; objects representing files, dates and times; or even user-defined objects.

Example (CLI)

Suppose you have a cluster named fred, consisting of nodes 101, 102, 103. Each node has a database tracking work requests with a table named request. You can find the total number of open requests in each database as follows (using the CLI):
    jao@zack$ osh @fred [ remote [ sql "select count(*) from request where state = 'open'" ] ] ^ out
    ('101', 1)
    ('102', 0)
    ('103', 5)

Example continued

Now suppose you want to find the total number of open requests across the cluster. You can pipe the (node, request count) tuples into an aggregation command:

    jao@zack$ osh @fred [ remote [ sql "select count(*) from request where state = 'open'" ] ] ^ agg 0 'total, node, count: total + count' $
    6
Note that this example combines remote execution on cluster nodes, database access (on each cluster node), and data processing (the aggregation step) in a single framework.

Example (API)

The same computation can be done using the API as follows:

    #!/usr/bin/python
    
    from osh.api import *
    
    osh(fork("fred",
             remote(sql("select count(*) from request where state = 'open'"))),
        agg(0, lambda total, node, count: total + count))        
More information:

License (GPL)
Release history
User's Guide
Command Reference Guide
Download
Software with similar goals
PyCon 2006 paper on osh
PyCon 2006 talk on osh
jao@geophile.com