Introduction
Hi, I am deploying a Python-Flask application on the Azure App Service for months.
Today I got the following error message while running my pipeline running on Microsoft DevOps:
ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/8d876019152941b/antenv/lib/python3.8/site-packages/numpy-1.18.1.dist-info'
I got the error on other apps stored on the same Plan. All my apps can't be updated anymore
what did I try
I created a new app and deployed the same code. I got the same error :
Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/REDACTED.zip (246.59 MB) to /tmp/zipdeploy/extracted
No space left on device
##[error]Failed to deploy web package to App Service.
The plan states that my threshold quota is 10 Gb and that I use 936Mb or 9% of my quota !
Using 'df-h' by using Kundu Bash
> Filesystem Size Used Avail Use% Mounted on
> none 29G 27G 0 100% /
> tmpfs 64M 0 64M 0% /dev
> tmpfs 953M 0 953M 0% /sys/fs/cgroup
> shm 62M 0 62M 0% /dev/shm
> //REDACTED/ 1000G 9.7G 991G 1% /home
> /dev/sda1 27G 8.6G 18G 34% /appsvctmp
> /dev/loop6p1 29G 27G 0 100% /etc/hosts
> udev 920M 0 920M 0% /dev/tty
> tmpfs 953M 0 953M 0% /proc/acpi
> tmpfs 953M 0 953M 0% /proc/scsi
> tmpfs 953M 0 953M 0% /sys/firmware
The question
What should I do to work again?
I can't access the /dev/loop6p1 directory (what is it?) and delete files
the Azure pipelines logs
> 2020-10-21 ##[section]Starting: Deploy Azure Web App : www.alt-f1.be
> 2020-10-21 ==============================================================================
> 2020-10-21 Task : Azure Web App
> 2020-10-21 Description : Deploy an Azure Web App for Linux or Windows
> 2020-10-21 Version : 1.168.3
> 2020-10-21 Author : Microsoft Corporation
> 2020-10-21 Help : https://aka.ms/azurewebapptroubleshooting
> 2020-10-21 ==============================================================================
> 2020-10-21 Got service connection details for Azure App Service:'www.alt-f1.be'
> 2020-10-21 Package deployment using ZIP Deploy initiated.
> 2020-10-21 Updating submodules.
> 2020-10-21 Preparing deployment for commit id '1ea14e5515'.
> 2020-10-21 Repository path is /tmp/zipdeploy/extracted
> 2020-10-21 Running oryx build...
> 2020-10-21 Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.8 -i /tmp/8d876019152941b -p compress_virtualenv=tar-gz -p virtualenv_name=antenv --log-file /tmp/build-debug.log
> 2020-10-21 Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
> 2020-10-21 You can report issues at https://github.com/Microsoft/Oryx/issues
> 2020-10-21
> 2020-10-21 Oryx Version: 0.2.20200917.1, Commit: 59deb778658a124cb74ea8e2c8f39fa87abcc9d9, ReleaseTagName: 20200917.1
> 2020-10-21
> 2020-10-21 Build Operation ID: |18pOsx/PQ8Y=.d0555319_
> 2020-10-21 Repository Commit : 1ea14e5515974a80bdcdc28eac749d4f
> 2020-10-21
> 2020-10-21 Detecting platforms...
> 2020-10-21 Detected following platforms:
> 2020-10-21 nodejs: 12.18.3
> 2020-10-21 python: 3.8.5
> 2020-10-21
> 2020-10-21 Using intermediate directory '/tmp/8d876019152941b'.
> 2020-10-21
> 2020-10-21 Copying files to the intermediate directory...
> 2020-10-21 Done in 6 sec(s).
> 2020-10-21
> 2020-10-21 Source directory : /tmp/8d876019152941b
> 2020-10-21 Destination directory: /home/site/wwwroot
> 2020-10-21
> 2020-10-21 Python Version: /opt/python/3.8.5/bin/python3.8
> 2020-10-21 Python Virtual Environment: antenv
> 2020-10-21 Creating virtual environment...
> 2020-10-21 Activating virtual environment...
> 2020-10-21 Running pip install...
> 2020-10-21 [20:41:33+0000] Collecting altf1be_helpers==2.1.4
> 2020-10-21 [20:41:33+0000] Downloading altf1be_helpers-2.1.4-py2.py3-none-any.whl (14 kB)
...
> 2020-10-21 [20:44:13+0000] Installing collected packages: numpy, six, python-dateutil, pytz, pandas, certifi, urllib3, idna, chardet, requests, isodate, oauthlib, requests-oauthlib, msrest, azure-core, pycparser, cffi, cryptography, azure-storage-blob, Unidecode, altf1be-helpers, attrs, Babel, pyparsing, packaging, > webencodings, bleach, cachelib, click, dash-core-components, dash-html-components, dash-table, MarkupSafe, Jinja2, Werkzeug, itsdangerous, Flask, brotli, Flask-Compress, retrying, plotly, dash-renderer, future, dash, docopt, docutils, Flask-Babel, Flask-Session, SQLAlchemy, Flask-SQLAlchemy, gunicorn, zipp, importlib-metadata, jeepney, > SecretStorage, keyring, more-itertools, yarg, pipreqs, pkginfo, pluggy, py, Pygments, pyodbc, pyopenssl, wcwidth, pytest, python-dotenv, readme-renderer, requests-toolbelt, starkbank-ecdsa, python-http-client, sendgrid, tqdm, twine, WTForms, xlrd, XlsxWriter
> 2020-10-21 ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/8d876019152941b/antenv/lib/python3.8/site-packages/numpy-1.18.1.dist-info'
> 2020-10-21
> 2020-10-21 WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
> 2020-10-21 You should consider upgrading via the '/tmp/8d876019152941b/antenv/bin/python -m pip install --upgrade pip' command.
> 2020-10-21 ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/8d876019152941b/antenv/lib/python3.8/site-packages/numpy-1.18.1.dist-info'\n\nWARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.\nYou should consider upgrading via the '/tmp/> 8d876019152941b/antenv/bin/python -m pip install --upgrade pip' command.\n/opt/Kudu/Scripts/starter.sh oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.8 -i /tmp/8d876019152941b -p compress_virtualenv=tar-gz -p virtualenv_name=antenv --log-file /tmp/build-debug.log
> 2020-10-21 ##[error]Failed to deploy web package to App Service.
> 2020-10-21 ##[error]To debug further please check Kudu stack trace URL : https://$www.alt-f1.be:***@www.alt-f1.be.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
> 2020-10-21 ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
> 2020-10-21 Successfully updated deployment History at https://www.alt-f1.be.scm.azurewebsites.net/api/deployments/2571603313079929
> 2020-10-21 App Service Application URL: https://www.alt-f1.be.azurewebsites.net
> 2020-10-21 ##[section]Finishing: Deploy Azure Web App : www.alt-f1.be
Conclusion
My plan is completely blocked. It is not possible to deploy any version of my code anymore.
@Abdelkrim, Microsoft-hosted agents provides 10 GB of storage for your source and build outputs- here - All the WebApps under Azure App Service plan shares resources.
If your source code and build outputs are more then 10GB, I suggest you to run your build using the Self Hosted Agent. For the python storage issue, you may try clearing the cache using the below command once and see if that helps. --> pip install --no-cache-dir tensorflow-gpu
Also just to isolate, the error is likely for hitting the temp (in your table shows 100% usage) -This a partition separate from your disk space. It is used for docker operations like unpacking/writing that require a temporary directory to offload files during processing. -When the integrated pipeline is used, it goes though webengine code which doesn't support more than 2 GB content-length.
In regards to drive access: - Looks like you're reviewing the App Service Plan storage space vs Hosted agents limitation. Also, Linux apps in App Service run in their own containers. No access to the host operating system is allowed, you do have root access to the container. Likewise, for apps running in Windows containers, you have administrative access to the container but no access to the host operating system.
<<Added as an answer;from comments>>