Skip to content

Commit bd44846

Browse files
author
Dan
committed
Added tests for API refactor. Updated readme.
1 parent f1a6dfe commit bd44846

File tree

3 files changed

+122
-65
lines changed

3 files changed

+122
-65
lines changed

‎README.rst‎

Lines changed: 28 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,9 @@ Installation
3131

3232
$ pip install parallel-ssh
3333

34-
**************
35-
Usage Examples
36-
**************
34+
*************
35+
Usage Example
36+
*************
3737

3838
See documentation on `read the docs`_ for more complete examples.
3939

@@ -42,15 +42,33 @@ Run `ls` on two remote hosts in parallel.
4242
>>> from pssh import ParallelSSHClient
4343
>>> hosts = ['myhost1', 'myhost2']
4444
>>> client = ParallelSSHClient(hosts)
45-
>>> output = client.run_command('ls -ltrh /tmp/aasdfasdf', sudo=True)
46-
>>> for host in output: print output
47-
[localhost] drwxr-xr-x 6 xxx xxx 4.0K Jan 1 00:00 xxx
48-
{'localhost': {'exit_code': 0, 'stdout': <generator>, 'stderr': <generator>, 'channel': channel}}
45+
>>> output = client.run_command('ls -ltrh /tmp/', sudo=True)
46+
>>> print output
47+
{'myhost1': {'exit_code': 0, 'stdout': <generator>, 'stderr': <generator>, 'channel': <channel>, 'cmd' : <greenlet>},
48+
'myhost2': {'exit_code': 0, 'stdout': <generator>, 'stderr': <generator>, 'channel': <channel>, 'cmd' : <greenlet>}}
49+
50+
Stdout and stderr buffers are available in output. Iterating on them can be used to get output as it becomes available.
51+
52+
>>> for host in output:
53+
>>> for line in output[host]['stdout']:
54+
>>> print "Host %s - output: %s" % (host, line)
55+
Host myhost1 - output: drwxr-xr-x 6 xxx xxx 4.0K Jan 1 00:00 xxx
56+
Host myhost2 - output: drwxr-xr-x 6 xxx xxx 4.0K Jan 1 00:00 xxx
4957

5058
**************************
5159
Frequently asked questions
5260
**************************
5361

62+
:Q:
63+
Why should I use this module and not, for example, `fabric <https://github.com/fabric/fabric>`_?
64+
65+
:A:
66+
Fabric is a port of `Capistrano <https://github.com/capistrano/capistrano>`_ from ruby to python. Its design goals are to provide a faithful port of capistrano with its `tasks` and `roles` to python with interactive command line being the intended usage. Its use as a library is non-standard and in `many <https://github.com/fabric/fabric/issues/521>`_ `cases <https://github.com/fabric/fabric/pull/674>`_ `just <https://github.com/fabric/fabric/pull/1215>`_ `plain <https://github.com/fabric/fabric/issues/762>`_ `broken <https://github.com/fabric/fabric/issues/1068>`_.
67+
68+
Furthermore, its parallel commands use a combination of both threads and processes with extremely high CPU usage and system load while running. Fabric currently stands at over 6,000 lines of code, majority of which is untested, particularly if used as a library as opposed to less than 700 lines of code currently in `ParallelSSH` with over 80% code test coverage.
69+
70+
ParallelSSH's design goals and motivation are to provide a *library* for running *asynchronous* SSH commands in parallel with **no** load induced on the system by doing so with the intended usage being completely programmatic and non-interactive - Fabric provides none of these goals.
71+
5472
:Q:
5573
Are SSH agents used?
5674

@@ -81,22 +99,14 @@ Frequently asked questions
8199
Yes, use the `pkey` parameter of the `ParallelSSHClient class <http://parallel-ssh.readthedocs.org/en/latest/#pssh.ParallelSSHClient>`_. For example:
82100

83101
>>> import paramiko
84-
>>> my_key = paramiko.RSAKey.from_private_key_file(my_rsa_key)
85-
>>> client = ParallelSSHClient(pkey=my_key)
86-
87-
:Q:
88-
Why should I use this module and not, for example, `fabric <https://github.com/fabric/fabric>`_?
89-
90-
:A:
91-
Fabric is a port of `capistrano <https://github.com/capistrano/capistrano>`_ from ruby to python. Its design goals are to provide a faithful port of capistrano with capistrano's `tasks` and `roles` to python with interactive command line being the intended usage - its use as a library is non-standard and in many cases just plain broken.
92-
Furthermore, its parallel commands use a combination of both threads and processes with extremely high CPU usage while its running. Fabric currently stands at more than 130,000 lines of code, a large proportion of which is untested, particularly if used as a library as opposed to less than 700 currently in `ParallelSSH` with over 70% code test coverage.
93-
ParallelSSH's design goals are to provide a *library* for running *asynchronous* SSH commands with **minimal** load induced on the system by doing so with the inteded usage being completely programmatic and non-interactive - Fabric provides none of these goals.
102+
>>> client_key = paramiko.RSAKey.from_private_key_file('user.key')
103+
>>> client = ParallelSSHClient(['myhost1', 'myhost2'], pkey=client_key)
94104

95105
********
96106
SFTP/SCP
97107
********
98108

99-
SFTP is supported (scp version 2) natively, no scp command used.
109+
SFTP is supported (SCP version 2) natively, no `scp` command required.
100110

101111
For example to copy a local file to remote hosts in parallel
102112

‎pssh.py‎

Lines changed: 57 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -292,9 +292,9 @@ class ParallelSSHClient(object):
292292
"""Uses :mod:`pssh.SSHClient`, performs tasks over SSH on multiple hosts in \
293293
parallel.
294294
295-
Connections to hosts are established in parallel when ``exec_command`` is called,
295+
Connections to hosts are established in parallel when ``run_command`` is called,
296296
therefor any connection and/or authentication exceptions will happen on the
297-
call to ``exec_command`` and need to be caught."""
297+
call to ``run_command`` and need to be caught."""
298298

299299
def __init__(self, hosts,
300300
user=None, password=None, port=None, pkey=None,
@@ -334,38 +334,30 @@ def __init__(self, hosts,
334334
UnknownHostException, ConnectionErrorException
335335
>>> client = ParallelSSHClient(['myhost1', 'myhost2'])
336336
>>> try:
337-
>>> ... cmds = client.exec_command('ls -ltrh /tmp/aasdfasdf', sudo = True)
337+
>>> ... output = client.run_command('ls -ltrh /tmp/aasdfasdf', sudo=True)
338338
>>> except (AuthenticationException, UnknownHostException, ConnectionErrorException):
339339
>>> ... return
340-
>>> output = [client.get_stdout(cmd) for cmd in cmds]
340+
>>> # Commands have started executing at this point
341+
>>> # Exit code will probably not be available immediately
342+
>>> print output
343+
>>> {'myhost1': {'exit_code': None,
344+
'stdout' : <generator>,
345+
'stderr' : <generator>,
346+
'cmd' : <greenlet>,
347+
},
348+
'myhost2': {'exit_code': None,
349+
'stdout' : <generator>,
350+
'stderr' : <generator>,
351+
'cmd' : <greenlet>,
352+
}}
353+
>>> # Print output as it comes in.
354+
>>> for host in output: print output[host]['stdout']
341355
[myhost1] ls: cannot access /tmp/aasdfasdf: No such file or directory
342356
[myhost2] ls: cannot access /tmp/aasdfasdf: No such file or directory
343-
>>> print output
344-
[{'myhost1': {'exit_code': 2}}, {'myhost2': {'exit_code': 2}}]
345-
346-
**Example with returned stdout and stderr buffers**
347-
348-
>>> from pssh import ParallelSSHClient, AuthenticationException,\
349-
UnknownHostException, ConnectionErrorException
350-
>>> client = ParallelSSHClient(['myhost1', 'myhost2'])
351-
>>> try:
352-
>>> ... cmds = client.exec_command('ls -ltrh /tmp/aasdfasdf', sudo = True)
353-
>>> except (AuthenticationException, UnknownHostException, ConnectionErrorException):
354-
>>> ... return
355-
>>> output = [client.get_stdout(cmd, return_buffers=True) for cmd in cmds]
356-
>>> print output
357-
[{'myhost1': {'exit_code': 2,
358-
'stdout' : <generator object <genexpr>,
359-
'stderr' : <generator object <genexpr>,}},
360-
{'myhost2': {'exit_code': 2,
361-
'stdout' : <generator object <genexpr>,
362-
'stderr' : <generator object <genexpr>,}},
363-
]
364-
>>> for host_stdout in output:
365-
... for line in host_stdout[host_output.keys()[0]]['stdout']:
366-
... print line
367-
ls: cannot access /tmp/aasdfasdf: No such file or directory
368-
ls: cannot access /tmp/aasdfasdf: No such file or directory
357+
>>> # Retrieve exit code after commands have finished
358+
>>> # `get_exit_code` will return `None` if command has not finished
359+
>>> print client.get_exit_code(output[host])
360+
0
369361
370362
**Example with specified private key**
371363
@@ -381,8 +373,8 @@ def __init__(self, hosts,
381373
object's life. To close them, just `del` or reuse the object reference.
382374
383375
>>> client = ParallelSSHClient(['localhost'])
384-
>>> cmds = client.exec_command('ls -ltrh /tmp/aasdfasdf')
385-
>>> cmds[0].join()
376+
>>> output = client.run_command('ls -ltrh /tmp/aasdfasdf')
377+
>>> client.pool.join()
386378
387379
:netstat: ``tcp 0 0 127.0.0.1:53054 127.0.0.1:22 ESTABLISHED``
388380
@@ -407,27 +399,48 @@ def __init__(self, hosts,
407399
self.host_clients = dict((host, None) for host in hosts)
408400

409401
def run_command(self, *args, **kwargs):
410-
"""Run command on all hosts in parallel, honoring self.pool_size
402+
"""Run command on all hosts in parallel, honoring self.pool_size,
403+
and return output buffers. This function will block until all commands
404+
have **started** and then return immediately. Any connection and/or
405+
authentication exceptions will be raised here and need catching.
411406
412-
:param args: Position arguments for command
407+
:param args: Positional arguments for command
413408
:type args: tuple
414409
:param kwargs: Keyword arguments for command
415410
:type kwargs: dict
416411
417-
:rtype: List of :mod:`gevent.Greenlet`
412+
:rtype: Dictionary with host as key as per :mod:`ParallelSSH.get_output`:
413+
``{'myhost1': {'exit_code': exit code if ready else None,
414+
'channel' : SSH channel of command,
415+
'stdout' : <iterable>,
416+
'stderr' : <iterable>,
417+
'cmd' : <greenlet>}}``
418418
419419
**Example**:
420420
421-
>>> output = client.exec_command('ls -ltrh')
421+
>>> output = client.run_command('ls -ltrh')
422+
423+
print stdout for each command:
424+
425+
>>> for host in output:
426+
>>> for line in output[host]['stdout']: print line
427+
428+
Get exit code after command has finished:
429+
430+
>>> for host in output:
431+
>>> for line in output[host]['stdout']: print line
432+
>>> exit_code = client.get_exit_code(output[host])
422433
423434
Wait for completion, no stdout:
424435
425436
>>> client.pool.join()
426437
427-
Alternatively/in addition print stdout for each command:
428-
438+
Capture stdout - **WARNING** - this will store the entirety of stdout
439+
into memory and may exhaust available memory if command output is
440+
large enough:
429441
>>> for host in output:
430-
>>> print output[host]['stdout']
442+
>>> stdout = list(output[host]['stdout'])
443+
>>> print "Complete stdout for host %s is %s" % (host, stdout,)
431444
"""
432445
for host in self.hosts:
433446
self.pool.spawn(self._exec_command, host, *args, **kwargs)
@@ -501,14 +514,16 @@ def get_output(self, commands=None):
501514
:rtype: Dictionary with host as key as in:
502515
``{'myhost1': {'exit_code': exit code if ready else None,
503516
'channel' : SSH channel of command,
504-
'stdout' : <iterable>,
505-
'stderr' : <iterable>,}}``"""
517+
'stdout' : <iterable>,
518+
'stderr' : <iterable>,
519+
'cmd' : <greenlet>}}``"""
506520
if not commands:
507521
commands = list(self.pool.greenlets)
508522
return {host: {'exit_code': self._get_exit_code(channel),
509523
'channel' : channel,
510524
'stdout' : stdout,
511-
'stderr' : stderr, }
525+
'stderr' : stderr,
526+
'cmd' : cmd, }
512527
for cmd in commands
513528
for (channel, host, stdout, stderr) in [cmd.get()]}
514529

‎tests/test_pssh_client.py‎

Lines changed: 37 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,8 @@ def tearDown(self):
5858
del self.listen_socket
5959

6060
def test_pssh_client_exec_command(self):
61-
server = start_server({ self.fake_cmd : self.fake_resp }, self.listen_socket)
61+
server = start_server({ self.fake_cmd : self.fake_resp },
62+
self.listen_socket)
6263
client = ParallelSSHClient(['127.0.0.1'], port=self.listen_port,
6364
pkey=self.user_key)
6465
cmd = client.exec_command(self.fake_cmd)[0]
@@ -70,7 +71,8 @@ def test_pssh_client_exec_command(self):
7071
server.join()
7172

7273
def test_pssh_client_exec_command_get_buffers(self):
73-
server = start_server({ self.fake_cmd : self.fake_resp }, self.listen_socket)
74+
server = start_server({ self.fake_cmd : self.fake_resp },
75+
self.listen_socket)
7476
client = ParallelSSHClient(['127.0.0.1'], port=self.listen_port,
7577
pkey=self.user_key)
7678
cmd = client.exec_command(self.fake_cmd)[0]
@@ -97,11 +99,40 @@ def test_pssh_client_exec_command_get_buffers(self):
9799
server.join()
98100

99101
def test_pssh_client_run_command_get_output(self):
100-
server = start_server({ self.fake_cmd : self.fake_resp }, self.listen_socket)
102+
server = start_server({ self.fake_cmd : self.fake_resp },
103+
self.listen_socket)
101104
client = ParallelSSHClient(['127.0.0.1'], port=self.listen_port,
102105
pkey=self.user_key)
103106
output = client.run_command(self.fake_cmd)
104-
# import ipdb; ipdb.set_trace()
107+
expected_exit_code = 0
108+
expected_stdout = [self.fake_resp]
109+
expected_stderr = []
110+
exit_code = output['127.0.0.1']['exit_code']
111+
stdout = list(output['127.0.0.1']['stdout'])
112+
stderr = list(output['127.0.0.1']['stderr'])
113+
self.assertEqual(expected_exit_code, exit_code,
114+
msg = "Got unexpected exit code - %s, expected %s" %
115+
(exit_code,
116+
expected_exit_code,))
117+
self.assertEqual(expected_stdout, stdout,
118+
msg = "Got unexpected stdout - %s, expected %s" %
119+
(stdout,
120+
expected_stdout,))
121+
self.assertEqual(expected_stderr, stderr,
122+
msg = "Got unexpected stderr - %s, expected %s" %
123+
(stderr,
124+
expected_stderr,))
125+
del client
126+
server.join()
127+
128+
def test_pssh_client_run_command_get_output_explicit(self):
129+
server = start_server({ self.fake_cmd : self.fake_resp },
130+
self.listen_socket)
131+
client = ParallelSSHClient(['127.0.0.1'], port=self.listen_port,
132+
pkey=self.user_key)
133+
out = client.run_command(self.fake_cmd)
134+
cmds = [cmd for host in out for cmd in [out[host]['cmd']]]
135+
output = client.get_output(commands=cmds)
105136
expected_exit_code = 0
106137
expected_stdout = [self.fake_resp]
107138
expected_stderr = []
@@ -177,7 +208,8 @@ def test_pssh_client_timeout(self):
177208
def test_pssh_client_exec_command_password(self):
178209
"""Test password authentication. Fake server accepts any password
179210
even empty string"""
180-
server = start_server({ self.fake_cmd : self.fake_resp }, self.listen_socket)
211+
server = start_server({ self.fake_cmd : self.fake_resp },
212+
self.listen_socket)
181213
client = ParallelSSHClient(['127.0.0.1'], port=self.listen_port,
182214
password='')
183215
cmd = client.exec_command(self.fake_cmd)[0]

0 commit comments

Comments
 (0)