Paramiko not receiving all data from Channel

Question

I've done a script to configure a couple of ASA devices. It does the job perfectly BUT I can't get the whole output from the devices I'm configuring, at some point the data gets stucked and there's no more output. I want to have that in order to check for problems or misconfigurations, etc. I'm configuring around 500 IPs, objects, groups, etc. from files on ASA firewalls... I don't know what to do, I haven't found any command to clean or erase Paramiko's buffer :(

Any ideas? This is my code:

import paramiko
import re
import time
from tqdm.auto import tqdm
from io import StringIO

device_ip = 'X.X.X.X'
ssh = paramiko.SSHClient()                               # Connection
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname = device_ip, username = username, password = password) 
connection = ssh.invoke_shell()
output_data = StringIO()
connection.send("conf t\n")                              # Enter configuration mode
time.sleep(1)                                         
file = open(file_name, 'r')                        # IP list file
lines = file.readlines()
objects = []
ip_subnet = re.compile(r'([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\/([0-9]+)')
group_name = "GRP-OBJ-EU-Blablabla"
for line in tqdm(lines):
    match = ip_subnet.findall(line.strip())
    ip = match[0][0]
    subnet = match[0][1]                               # Not used, for future use may be...
    object_name = "obj-"+ip
    object_configuration = "object network "+object_name
    object_network = "host "+ip
    object_description = "description whatever"
    objects.append(object_name)
    connection.send(object_configuration+"\n")
    time.sleep(1)
    connection.send(object_network+"\n")
    time.sleep(1)
    connection.send(object_description+"\n")
    time.sleep(1)
    received_data = connection.recv(5000).decode(encoding='utf-8')
    if received_data:
        output_data.write(received_data)
group_command = "object-group network "+group_name
connection.send(group_command+"\n")
time.sleep(1)
for object_host in tqdm(objects):
    connection.send("network-object object "+object_host+"\n")
    time.sleep(1)
    received_data = connection.recv(5000).decode(encoding='utf-8')
    if received_data:
        output_data.write(received_data)
connection.send("end \n")
time.sleep(1)
connection.send("wr \n")
time.sleep(5)
connection.close()
ssh.close()
file.close()
print(output_data)

I've tried a line like this one below but it does not work either:

device_output = connection.recv(1000000000000).decode(encoding='utf-8')

Answer 1

Well I've found the solution... it's pretty dumb. I was not exiting enable mode... and it seems that was it! lol I don't get the relation but anyways the lines below work... Thanks everyone for your help!

....
connection.send("end \n")
time.sleep(1)
connection.send("wr \n")
time.sleep(5)
device_output = connection.recv(10000000).decode(encoding='utf-8') 
connection.close()
ssh.close()
file.close()
print(device_output)

Answer 2

I could not directly test your code, but I had a similar problem in the past and found a practical solution. To get all the data at once, you can open a notepad and print the received_data data in it. You can also use ascii outside of the encoding='utf-8' method.

Example Code:

received_data = remote_connection.recv(99999999)
result = received_data.decode('ascii').strip("\n")

LOGfile = "log.txt" 
log = open(LOGfile, 'a')
log.write(result )
log.close()

I recommend Netmiko-TTP library to parse all the data you want except the Paramiko library. I have given a sample code link below. I hope that will be useful.

https://github.com/MertKulac/Nokia--TTP--Template--Command--Set/blob/main/show%20system%20informtaion.py