GCE × CentOS8 で Jupyter 環境を構築する
タイトルのままですが、CentOS8 が使えたので、勉強がてら触ってみました。 基本的には Docker および docker-compose を使用してコンテナの Jupyter 環境を構築します。
docker のインストール
どんな方法でもいいので、GCP 上でインスタンスを立ててください。 その際に CentOS8 のイメージにするのを忘れずに。
SSH でコンソールに繋いだところからはじめます。
まずは yum update の代わり。
[user@jupyter ~]$ sudo su -
[root@jupyter ~]# dnf -y update CentOS-8 - AppStream 1.5 MB/s | 5.6 MB 00:03 CentOS-8 - Base 2.0 MB/s | 5.3 MB 00:02 CentOS-8 - Extras 1.5 kB/s | 2.1 kB 00:01 Google Cloud SDK 5.9 MB/s | 27 MB 00:04 Google Compute Engine 1.6 kB/s | 5.7 kB 00:03 Dependencies resolved. =================================================================================================================== Package Arch Version Repository Size =================================================================================================================== Upgrading: google-cloud-sdk noarch 266.0.0-1 google-cloud-sdk 35 M Transaction Summary =================================================================================================================== Upgrade 1 Package Total download size: 35 M Downloading Packages: 8ad4cecdf8d21424d0e49bca00836de86cb608625384064f5d6a0c73eae2df16-google-cloud-sdk- 12 MB/s | 35 MB 00:02 ------------------------------------------------------------------------------------------------------------------- Total 12 MB/s | 35 MB 00:02 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Upgrading : google-cloud-sdk-266.0.0-1.noarch 1/2 Running scriptlet: google-cloud-sdk-266.0.0-1.noarch 1/2 Running scriptlet: google-cloud-sdk-265.0.0-1.noarch 2/2 Cleanup : google-cloud-sdk-265.0.0-1.noarch 2/2 Running scriptlet: google-cloud-sdk-265.0.0-1.noarch 2/2 Verifying : google-cloud-sdk-266.0.0-1.noarch 1/2 Verifying : google-cloud-sdk-265.0.0-1.noarch 2/2 Upgraded: google-cloud-sdk-266.0.0-1.noarch Complete!
installの前準備
[root@jupyter ~]# dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[root@jupyter ~]# dnf repolist Docker CE Stable - x86_64 3.0 kB/s | 3.5 kB 00:01 Google Cloud SDK 195 B/s | 454 B 00:02 Google Compute Engine 199 B/s | 454 B 00:02 repo id repo name status AppStream CentOS-8 - AppStream 4,928 BaseOS CentOS-8 - Base 2,713 docker-ce-stable Docker CE Stable - x86_64 53 extras CentOS-8 - Extras 3 google-cloud-sdk Google Cloud SDK 948 google-compute-engine Google Compute Engine 6
なぜかそのままinstallができないので 別バージョンを指定
[root@jupyter ~]# dnf -y install wget Last metadata expiration check: 0:02:03 ago on Fri 11 Oct 2019 01:56:03 AM UTC. Dependencies resolved. =================================================================================================================== Package Arch Version Repository Size =================================================================================================================== Installing: wget x86_64 1.19.5-7.el8_0.1 AppStream 734 k Transaction Summary =================================================================================================================== Install 1 Package Total download size: 734 k Installed size: 2.8 M Downloading Packages: wget-1.19.5-7.el8_0.1.x86_64.rpm 608 kB/s | 734 kB 00:01 --- Total 318 kB/s | 734 kB 00:02 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : wget-1.19.5-7.el8_0.1.x86_64 1/1 Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64 1/1 Verifying : wget-1.19.5-7.el8_0.1.x86_64 1/1 Installed: wget-1.19.5-7.el8_0.1.x86_64 Complete!
[root@jupyter ~]# wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.2-3.3.el7.x86_64.rpm --2019-10-11 01:58:53-- https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.2-3.3. el7.x86_64.rpm Resolving download.docker.com (download.docker.com)... 13.249.87.50, 13.249.87.47, 13.249.87.20, ... Connecting to download.docker.com (download.docker.com)|13.249.87.50|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 23159364 (22M) [binary/octet-stream] Saving to: ‘containerd.io-1.2.2-3.3.el7.x86_64.rpm’ containerd.io-1.2.2-3.3.el7. 100%[=============================================>] 22.09M 86.2MB/s in 0.3s 2019-10-11 01:58:53 (86.2 MB/s) - ‘containerd.io-1.2.2-3.3.el7.x86_64.rpm’ saved [23159364/23159364][root@jupyter ~]#
[root@jupyter ~]# dnf remove containerd.io No match for argument: containerd.io No packages marked for removal. Dependencies resolved. Nothing to do. Complete!
[root@jupyter ~]# dnf install containerd.io-1.2.2-3.3.el7.x86_64.rpm Last metadata expiration check: 0:01:55 ago on Fri 11 Oct 2019 02:26:59 AM UTC. Dependencies resolved. ============================================================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================================================= Installing: containerd.io x86_64 1.2.2-3.3.el7 @commandline 22 M # Transaction Summary Install 1 Package Total size: 22 M Installed size: 88 M Is this ok [y/N]: y Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : containerd.io-1.2.2-3.3.el7.x86_64 1/1 Running scriptlet: containerd.io-1.2.2-3.3.el7.x86_64 1/1 Verifying : containerd.io-1.2.2-3.3.el7.x86_64 1/1 Installed: containerd.io-1.2.2-3.3.el7.x86_64 Complete!
[root@jupyter ~]# dnf install -y docker-ce docker-ce-cli Last metadata expiration check: 0:04:01 ago on Fri 11 Oct 2019 01:56:03 AM UTC. Dependencies resolved. =================================================================================================================== Package Arch Version Repository Size =================================================================================================================== Installing: docker-ce x86_64 3:19.03.3-3.el7 docker-ce-stable 24 M docker-ce-cli x86_64 1:19.03.3-3.el7 docker-ce-stable 39 M Installing dependencies: container-selinux noarch 2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7 AppStream 43 k libcgroup x86_64 0.41-19.el8 BaseOS 70 k policycoreutils-python-utils noarch 2.8-16.1.el8 BaseOS 228 k Enabling module streams: container-tools rhel8 Transaction Summary =================================================================================================================== Install 5 Packages Total download size: 64 M Installed size: 273 M Downloading Packages: (1/5): libcgroup-0.41-19.el8.x86_64.rpm 62 kB/s | 70 kB 00:01 (2/5): container-selinux-2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch.rpm 39 kB/s | 43 kB 00:01 (3/5): policycoreutils-python-utils-2.8-16.1.el8.noarch.rpm 187 kB/s | 228 kB 00:01 (4/5): docker-ce-19.03.3-3.el7.x86_64.rpm 41 MB/s | 24 MB 00:00 (5/5): docker-ce-cli-19.03.3-3.el7.x86_64.rpm 45 MB/s | 39 MB 00:00 --- Total 6.4 MB/s | 64 MB 00:10 warning: /var/cache/dnf/docker-ce-stable-091d8a9c23201250/packages/docker-ce-19.03.3-3.el7.x86_64.rpm: Header V4 RS A/SHA512 Signature, key ID 621e9f35: NOKEY Docker CE Stable - x86_64 1.4 kB/s | 1.6 kB 00:01 Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35 From : https://download.docker.com/linux/centos/gpg Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : docker-ce-cli-1:19.03.3-3.el7.x86_64 1/5 Running scriptlet: docker-ce-cli-1:19.03.3-3.el7.x86_64 1/5 Installing : policycoreutils-python-utils-2.8-16.1.el8.noarch 2/5 Installing : container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch 3/5 Running scriptlet: container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch 3/5 Running scriptlet: libcgroup-0.41-19.el8.x86_64 4/5 Installing : libcgroup-0.41-19.el8.x86_64 4/5 Running scriptlet: libcgroup-0.41-19.el8.x86_64 4/5 Installing : docker-ce-3:19.03.3-3.el7.x86_64 5/5 Running scriptlet: docker-ce-3:19.03.3-3.el7.x86_64 5/5 Verifying : container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch 1/5 Verifying : libcgroup-0.41-19.el8.x86_64 2/5 Verifying : policycoreutils-python-utils-2.8-16.1.el8.noarch 3/5 Verifying : docker-ce-3:19.03.3-3.el7.x86_64 4/5 Verifying : docker-ce-cli-1:19.03.3-3.el7.x86_64 5/5 Installed: docker-ce-3:19.03.3-3.el7.x86_64 docker-ce-cli-1:19.03.3-3.el7.x86_64 container-selinux-2:2.94-1.git1e99f1d.module_el8.0.0+58+91b614e7.noarch libcgroup-0.41-19.el8.x86_64 policycoreutils-python-utils-2.8-16.1.el8.noarch Complete!
[root@jupyter ~]# docker --version Docker version 19.03.3, build a872fc2f86
[root@jupyter ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
入りましたね。
自動起動設定をしておく
[root@jupyter ~]# systemctl start docker
[root@jupyter ~]# systemctl enable docker Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service.
docker-composeは入っていない状態。
[root@jupyter ~]# docker-compose -bash: docker-compose: command not found
dnfもない。
[root@jupyter ~]# dnf -y install docker-compose Last metadata expiration check: 0:07:38 ago on Fri 11 Oct 2019 01:56:03 AM UTC. No match for argument: docker-compose Error: Unable to find a match
公式の通りにいれる
[root@jupyter ~]# curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 617 0 617 0 0 2924 0 --:--:-- --:--:-- --:--:-- 2910 100 15.4M 100 15.4M 0 0 20.9M 0 --:--:-- --:--:-- --:--:-- 20.9M
[root@jupyter ~]# chmod +x /usr/local/bin/docker-compose
[root@jupyter ~]# ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
[root@jupyter ~]# docker-compose --version docker-compose version 1.24.1, build 4667896b
うまく入った。
Jupyterコンテナの準備
Dockerの準備に比べて、やることは少ないです。 juyterのDockerfileを設置したいディレクトリに移動し、 下記のリポジトリをクローンします。
dnf -y install git
git clone https://github.com/hiroyannnn/jupyter.git
cloneするのが嫌な場合は、リポジトリ内のファイルをviなどで作成しても良いです。
cloneした場合、jupyterというディレクトリができるので、cdします。
cd jupyter
configファイルにパスフレーズを入れているので、 opensslなどでパスフレーズを作ってください。
sha1:d4a8b1853fdf:(...省略...)fa181ec59716
みたいなやつです。
# create your key # and modify config file.
いれる場所はここ
c.NotebookApp.password = ''
Dockerを起動します。
docker-compose up
コンソールにログが出力されるので、 起動したっぽいログがでたら、 パブリックIP:8888 でアクセスできるはず。 暗号化したパスフレーズを入力してログインしてください。
データはdata
ディレクトリに入ります。
ついでに、ホスト側のdataは、 GCEの追加diskをマウントしておくと、データ量が増えた時も調整しやすいです。
参考
https://qiita.com/dora_2562/items/24691d3bec4c99c1d794 https://qiita.com/y4m3/items/c2703d4e131e05084b7b
余談
ちなみに一回下記でコマンドミスして
# 本来はこっち # dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo # 間違えてliで改行が入っちゃった dnf config-manager --add-repo https://download.docker.com/li
repolistが失敗するようになりましたが
[root@jupyter ~]# dnf repolist CentOS-8 - AppStream 1.8 kB/s | 4.3 kB 00:02 CentOS-8 - Base 1.6 kB/s | 3.8 kB 00:02 CentOS-8 - Extras 1.1 kB/s | 1.5 kB 00:01 created by dnf config-manager from https://download.docker.com/li 456 B/s | 550 B 00:01 Error: Failed to synchronize cache for repo 'download.docker.com_li'
そのファイルを消しちゃえばリカバリ可能でした。
[root@jupyter ~]# ls -l /etc/yum.repos.d/ total 56 -rw-r--r--. 1 root root 731 Aug 14 06:42 CentOS-AppStream.repo -rw-r--r--. 1 root root 712 Aug 14 06:42 CentOS-Base.repo -rw-r--r--. 1 root root 798 Aug 14 06:42 CentOS-centosplus.repo -rw-r--r--. 1 root root 1320 Aug 14 06:42 CentOS-CR.repo -rw-r--r--. 1 root root 668 Aug 14 06:42 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 756 Aug 14 06:42 CentOS-Extras.repo -rw-r--r--. 1 root root 338 Aug 14 06:42 CentOS-fasttrack.repo -rw-r--r--. 1 root root 928 Aug 14 06:42 CentOS-Media.repo -rw-r--r--. 1 root root 736 Aug 14 06:42 CentOS-PowerTools.repo -rw-r--r--. 1 root root 1382 Aug 14 06:42 CentOS-Sources.repo -rw-r--r--. 1 root root 74 Aug 14 06:42 CentOS-Vault.repo -rw-r--r--. 1 root root 2424 Oct 11 01:43 docker-ce.repo -rw-r--r--. 1 root root 145 Oct 11 01:43 download.docker.com_li.repo -rw-r--r--. 1 root root 591 Oct 2 22:49 google-cloud.repo
[root@jupyter ~]# rm /etc/yum.repos.d/download.docker.com_li.repo rm: remove regular file '/etc/yum.repos.d/download.docker.com_li.repo'? y
[root@jupyter ~]# dnf repolist Docker CE Stable - x86_64 3.0 kB/s | 3.5 kB 00:01 Google Cloud SDK 195 B/s | 454 B 00:02 Google Compute Engine 199 B/s | 454 B 00:02 repo id repo name status AppStream CentOS-8 - AppStream 4,928 BaseOS CentOS-8 - Base 2,713 docker-ce-stable Docker CE Stable - x86_64 53 extras CentOS-8 - Extras 3 google-cloud-sdk Google Cloud SDK 948 google-compute-engine Google Compute Engine 6 [root@jupyter ~]#