I would like to use Google Cloud SDK (including the Python Extensions for App Engine) inside Compute Engine, so I can replicate my local development setup in a virtual machine. I.e. run local dev_appserver.py and unit-testing on the VM, or deploy new app versions to Google App Engine.
After creating a new VM instance from the default Ubuntu 16.04 image and (machine type n1-standard-1), I have noticed that gcloud is already pre-installed.
anh@ani-dev-2:~$ gcloud components list
Your current Cloud SDK version is: 163.0.0
The latest available version is: 169.0.0
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Components │
├──────────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬───────────┤
│ Status │ Name │ ID │ Size │
├──────────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼───────────┤
│ Update Available │ BigQuery Command Line Tool │ bq │ < 1 MiB │
│ Update Available │ Cloud SDK Core Libraries │ core │ 6.7 MiB │
│ Update Available │ Cloud Storage Command Line Tool │ gsutil │ 3.0 MiB │
│ Not Installed │ App Engine Go Extensions │ app-engine-go │ 98.0 MiB │
│ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 4.1 MiB │
│ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 3.5 MiB │
│ Not Installed │ Cloud Datalab Command Line Tool │ datalab │ < 1 MiB │
│ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 15.4 MiB │
│ Not Installed │ Cloud Datastore Emulator (Legacy) │ gcd-emulator │ 38.1 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 33.2 MiB │
│ Not Installed │ Emulator Reverse Proxy │ emulator-reverse-proxy │ 14.5 MiB │
│ Not Installed │ Google Container Local Builder │ container-builder-local │ 3.7 MiB │
│ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 2.2 MiB │
│ Not Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │
│ Not Installed │ gcloud Beta Commands │ beta │ < 1 MiB │
│ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 128.1 MiB │
│ Not Installed │ gcloud app Python Extensions │ app-engine-python │ 6.5 MiB │
│ Not Installed │ kubectl │ kubectl │ 16.0 MiB │
│ Installed │ Default set of gcloud commands │ gcloud │ │
└──────────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴───────────┘
Intuitively, I've tried to install the missing app-engine-python component.
anh@ani-dev-2:~$ gcloud components install app-engine-python
You cannot perform this action because this Cloud SDK installation is
managed by an external package manager. If you would like to get the
latest version, please see our main download page at:
https://cloud.google.com/sdk/
ERROR: (gcloud.components.install) The component manager is disabled for
this installation
As far as I understand, there are alternative installation methods available (yum, debian) and deb was used for the VM image, which is the reason the component manager is disabled:
anh@ani-dev-2:~$ dpkg -s google-cloud-sdk
Package: google-cloud-sdk
Status: install ok installed
Priority: optional
Section: partner/admin
Installed-Size: 93228
Maintainer: Canonical CPC Team <[email protected]>
Architecture: all
Version: 163.0.0-0ubuntu1~16.04.0
Depends: findutils, python2.7, python:any (>= 2.6.6-7~), python-crcmod, python-google-compute-engine
Conflicts: gsutils
Conffiles:
/etc/bash_completion.d/completion.bash.inc 860b91bfb6a0e6b513321ea6ca429e4c
Description: Google Cloud SDK for interacting with Google Cloud services.
The command-line tools for Google App Engine, Compute Engine,
Cloud Storage, BigQuery, Cloud SQL, and Cloud DNS are bundled as part of
the Cloud SDK.
.
This is a packaged distribution of the Google Cloud SDK. For up-to-date
versions please see https://developers.google.com/cloud/sdk.
Homepage: https://developers.google.com/cloud/sdk/
So I've installed successfully the missing component google-cloud-sdk-app-engine-python via apt-get as explained in the docs Installing Cloud SDK. It seems that the pre-installed Cloud SDK package 163.0.0 was replaced by the recent version 169.0.0, and the python component is now installed.
anh@ani-dev-2:~$ gcloud components list
Your current Cloud SDK version is: 169.0.0
The latest available version is: 169.0.0
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Components │
├───────────────┬──────────────────────────────────────────────────────┬──────────────────────────┬───────────┤
│ Status │ Name │ ID │ Size │
├───────────────┼──────────────────────────────────────────────────────┼──────────────────────────┼───────────┤
│ Not Installed │ App Engine Go Extensions │ app-engine-go │ 98.0 MiB │
│ Not Installed │ Cloud Bigtable Command Line Tool │ cbt │ 4.1 MiB │
│ Not Installed │ Cloud Bigtable Emulator │ bigtable │ 3.5 MiB │
│ Not Installed │ Cloud Datalab Command Line Tool │ datalab │ < 1 MiB │
│ Not Installed │ Cloud Datastore Emulator │ cloud-datastore-emulator │ 15.4 MiB │
│ Not Installed │ Cloud Datastore Emulator (Legacy) │ gcd-emulator │ 38.1 MiB │
│ Not Installed │ Cloud Pub/Sub Emulator │ pubsub-emulator │ 33.2 MiB │
│ Not Installed │ Emulator Reverse Proxy │ emulator-reverse-proxy │ 14.5 MiB │
│ Not Installed │ Google Container Local Builder │ container-builder-local │ 3.7 MiB │
│ Not Installed │ Google Container Registry's Docker credential helper │ docker-credential-gcr │ 2.2 MiB │
│ Not Installed │ gcloud app Java Extensions │ app-engine-java │ 128.1 MiB │
│ Not Installed │ kubectl │ kubectl │ 16.0 MiB │
│ Installed │ BigQuery Command Line Tool │ bq │ < 1 MiB │
│ Installed │ Cloud SDK Core Libraries │ core │ 6.7 MiB │
│ Installed │ Cloud Storage Command Line Tool │ gsutil │ 3.0 MiB │
│ Installed │ Default set of gcloud commands │ gcloud │ │
│ Installed │ gcloud Alpha Commands │ alpha │ < 1 MiB │
│ Installed │ gcloud Beta Commands │ beta │ < 1 MiB │
│ Installed │ gcloud app Python Extensions │ app-engine-python │ 6.5 MiB │
└───────────────┴──────────────────────────────────────────────────────┴──────────────────────────┴───────────┘
Is this approach the recommended way to add Cloud SDK components on Compute Engine?
Is the same approach also supported with a micro or small machine-type? I assume,
f1-micro(org1-small) would be sufficient for my purpose. But maybe since in the smaller types the VM shares the CPU I wouldn't have permissions to change the installation. If this is the case, am I supposed to install an additional, custom Cloud SDK manually into my user folder, e.g.$HOME/google-cloud-sdklike I did on my Mac?
P.S. 1
This is the output when I perform the same steps in the f1-micro (same VM image), I've tried twice on two new instances:
anh@ani-dev-1:~$ sudo apt-get update && sudo apt-get install google-cloud-sdk
Hit:1 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:4 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial/main Sources [868 kB]
Get:5 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial/restricted Sources [4,808 B]
Get:6 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial/universe Sources [7,728 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:8 http://archive.canonical.com/ubuntu xenial InRelease
Get:9 http://packages.cloud.google.com/apt cloud-sdk-xenial InRelease [6,327 B]
Get:10 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB]
Get:11 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/main Sources [273 kB]
Get:12 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [3,400 B]
Get:13 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/universe Sources [171 kB]
Get:14 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [7,232 B]
Get:15 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [632 kB]
Get:16 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [261 kB]
Get:17 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [8,048 B]
Get:18 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2,688 B]
Get:19 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [529 kB]
Get:20 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [209 kB]
Get:21 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-backports/main Sources [3,396 B]
Get:22 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-backports/universe Sources [4,404 B]
Get:23 http://us-east1.gce.archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [4,884 B]
Get:24 http://packages.cloud.google.com/apt cloud-sdk-xenial/main amd64 Packages [9,625 B]
Get:25 http://security.ubuntu.com/ubuntu xenial-security/main Sources [90.1 kB]
Get:26 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2,604 B]
Get:27 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [40.9 kB]
Get:28 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [1,144 B]
Get:29 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [353 kB]
Get:30 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [154 kB]
Get:31 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [166 kB]
Get:32 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [86.8 kB]
Fetched 12.1 MB in 2s (4,922 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
google-cloud-sdk-app-engine-java google-cloud-sdk-app-engine-python google-cloud-sdk-pubsub-emulator
google-cloud-sdk-bigtable-emulator google-cloud-sdk-datastore-emulator kubectl
The following packages will be upgraded:
google-cloud-sdk
1 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
Need to get 13.8 MB of archives.
After this operation, 10.7 MB of additional disk space will be used.
Get:1 http://packages.cloud.google.com/apt cloud-sdk-xenial/main amd64 google-cloud-sdk all 169.0.0-0 [13.8 MB]
Fetched 13.8 MB in 1s (11.4 MB/s)
(Reading database ... 64568 files and directories currently installed.)
Preparing to unpack .../google-cloud-sdk_169.0.0-0_all.deb ...
Unpacking google-cloud-sdk (169.0.0-0) over (163.0.0-0ubuntu1~16.04.0) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up google-cloud-sdk (169.0.0-0) ...
Killed
dpkg: error processing package google-cloud-sdk (--configure):
subprocess installed post-installation script returned error exit status 137
E: Sub-process /usr/bin/dpkg returned an error code (1)
P.S. 2
It seems the reason why the installation process is killed in f1-micro is some kind of out-of-memory error. See:
https://serverfault.com/questions/862019/how-to-resolve-dpkg-error-processing-package-google-cloud-sdk
and https://groups.google.com/forum/#!topic/gce-discussion/BBHc2QebbBw
However, I can achieve my goal with a bigger machine-type.
Apt-get is the way to install the app engine component In a Ubuntu/Debían System.
If this is a recurring installation among your VM's you might want to write a start-up script to do this installation. Or save that finished installation as an image, depending on your booting time requirements.
About the shared processor, that is not going to affect at all your installation.