ft_services → Kubernetes Orchestration

쿠버네티스로 컨테이너 서비스 운용하기 (ft_services)

컨테이너와 컨테이너 오케스트레이션

쿠버네티스

쿠버네티스가 할 수 있는 것

• 로드 밸런싱 (Load Balancing / 부하 분산)
로드 밸런싱은 위에서 말한 접속자가 많을 때 혹은 적을 때 이에 따른 제어를 해 주는 것을 말한다. 로드 밸런싱은 쿠버네티스만의 기술은 아니다. 하지만 인터넷 서비스 특성상 로드 밸런싱 개념을 많이 사용하고 부하(트래픽)에 따라 쿠버네티스가 원활하게 자원을 제어해 준다.

• 자동 복구 (Self-healing)
특정 컨테이너가 오류로 종료되거나 하는 상황에서 알아서 해당 컨테이너를 복구해준다. 이런 과정들을 클라이언트가 모르는 새 수행하게 할 수 있다.

• 현재 사용환경에 따른 최적의 자원 분배 (bin packing)
컨테이너에 따라 필요로 하는 자원 사용량 등이 다를 것이다. 이를 컨테이너가 필요로 하는 만큼 적절히 할당하게 할 수 있다.

• 롤아웃/롤백
현재 배포중인 서비스를 이전으로 되돌리거나 신규 서비스를 배포하는 것을 말한다. 쿠버네티스로 이를 자동화 할 수 있다.

• 기타 여러가지 기능들

쿠버네티스 구성 요소

파드 (Pod)

레플리카 셋 (Replica set)

노드 (Node)

마스터 (Master)

Deployment (디플로이먼트)

쿠버네티스 클러스터

아주 기본적인 클러스터 생성해보기

minikube

minikube 설치 (macOS)

# 방법 1. brew 이용
$> brew install minikube
# 방법 2. minikube 프로그램 내려받아서 실행
$> curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64
$> chmod +x minikube

minikube 실행

$> minikube start
😄  Darwin 11.1 위의 minikube v1.17.0
✨  자동적으로 docker 드라이버가 선택되었습니다. 다른 드라이버 목록: hyperkit, ssh
👍  minikube 클러스터의 minikube 컨트롤 플레인 노드를 시작하는 중
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v8-v1....: 491.22 MiB / 491.22 MiB  100.00% 35.35 Mi
🔥  Creating docker container (CPUs=2, Memory=1988MB) ...
🐳  쿠버네티스 v1.20.2 을 Docker 20.10.2 런타임으로 설치하는 중
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$> docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS          PORTS                                                                                                      NAMES
e611616fe069   gcr.io/k8s-minikube/kicbase:v0.0.17   "/usr/local/bin/entr…"   15 minutes ago   Up 15 minutes   127.0.0.1:55007->22/tcp, 127.0.0.1:55006->2376/tcp, 127.0.0.1:55005->5000/tcp, 127.0.0.1:55004->8443/tcp   minikube

파드 생성 및 디플로이먼트 만들기

$> kubectl create deployment newnode --image=docker.io/library/nginx:alpine
deployment.apps/newnode created

$> kubectl get deployments
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
newnode   1/1     1            1           27s

$> kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
newnode-649dfbf6dc-fpqvh   1/1     Running   0          37s

서비스 생성

$> kubectl expose deployment newnode --type=LoadBalancer --port=80
service/newnode exposed

$> kubectl get services
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP        167m
newnode      LoadBalancer   10.104.145.214   <pending>     80:30577/TCP   15s

$> minikube service newnode
|-----------|---------|-------------|---------------------------|
| NAMESPACE |  NAME   | TARGET PORT |            URL            |
|-----------|---------|-------------|---------------------------|
| default   | newnode |          80 | http://192.168.49.2:30577 |
|-----------|---------|-------------|---------------------------|
🏃  newnode 서비스의 터널을 시작하는 중
|-----------|---------|-------------|------------------------|
| NAMESPACE |  NAME   | TARGET PORT |          URL           |
|-----------|---------|-------------|------------------------|
| default   | newnode |             | http://127.0.0.1:64353 |
|-----------|---------|-------------|------------------------|
🎉  Opening service default/newnode in default browser...
❗  Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

$> kubectl delete service newnode
service "newnode" deleted
$> kubectl delete deployment newnode
deployment.apps "newnode" deleted
$> minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1개의 노드가 중지되었습니다.
$> minikube stop
🔥  docker 의 "minikube" 를 삭제하는 중 ...
🔥  Deleting container "minikube" ...
🔥  /Users/joohongpark/.minikube/machines/minikube 제거 중...
💀  "minikube" 클러스터 관련 정보가 모두 삭제되었습니다

ft_service 요구사항 분석

목표

• 쿠버네티스를 이용해 클러스터를 관리하고 서비스를 배포함.

• 네트워크를 가상화하고 클러스터링 할 수 있어야 함.

• 각각 다른 서비스 (웹서버/데이터베이스 등)로 나눌수 있어야 함.

요구사항

• setup.sh 파일을 제외한 모든 설정파일들은 srcs 폴더 내에 있어야 하며 클러스터를 세팅할 수 있는 쉘 파일은 setup.sh 파일로 존재해야 한다.

• 클러스터 구성에 사용할 모든 이미지는 Dockerfile로 빌드가 되어야 한다.
• 도커 빌드에 사용되는 이미지는 알파인 리눅스 (https://hub.docker.com/_/alpine) 를 사용해야 한다.

• 쿠버네티스 관련
• 쿠버네티스 웹 대쉬보드에 접속하여 클러스터를 관리할 수 있어야 한다.
• Load Balancer를 이용해 외부에서 클러스터로 접속할 수 있어야 한다. (다른 경로로 클러스터에 진입할 수 있으면 안된다.)
• 로드 밸런서는 싱글 아이피를 사용한다.
• 워드프레스 관련
• 워드프레스 웹 사이트는 포트 5050에서 동작되어야 한다.
• 워드프레스와 MySQL 기반 데이터베이스는 서로 다른 컨테이너에서 동작되어야 한다.
• 워드프레스는 관리자를 포함한 몇 유저들이 생성되어 있어야 한다.
• 워드프레스는 워드프레스 구동을 위한 nginx 서버가 필요할 것이다.
• Load Balancer는 워드프레스로 리다이렉트 할 수 있어야 한다.
• phpmyadmin 관련
• phpmyadmin은 포트 5050에서 동작되어야 한다.
• phpmyadmin 구동을 위한 nginx 서버가 필요할 것이다.
• Load Balancer는 phpmyadmin으로 리다이렉트 할 수 있어야 한다.
• nginx 관련 (위 phpmyadmin/워드프레스 nginx와 다른듯)
• nginx 서버는 80, 440 포트에서 동작되어야 한다.
• 80 포트는 HTTP 301을 이용해 443으로 리다이렉션 해야 한다.
• nginx 서버는 아무거나 표시 가능하다. (에러만 아니면 됨)
• /wordpress로 접근할 시 HTTP 307을 이용해 워드프레스 아이피:포트로 접속할수 있도록 함.
• /phpmyadmin으로 접근할 시 리버스 프록시를 이용해 워드프레스 아이피:포트로 접속할수 있도록 함.
• FTPS(파일 전송 프로토콜)는 21번 포트로 동작하도록 함.
• Grafana 관련
• 포트 3000에서 동작하도록 함.
• InfluxDB에 링크되어 있어야 함.
• 이게 모든 컨테이너를 모니터링 할 것임.
• 각 서비스에 대한 대쉬보드를 만들어야 함.
• InfluxDB와 Grafana는 서로 다른 컨테이너 내에 있어야 함.
• 기타
• 두개의 데이터베이스 중 하나가 정지된다면 데이터가 유지가 되어야 함.
• nginx 컨테이너에 SSH로 로그인 할 수 있어야 함.
• 컴포넌트 중 하나가 정지된다면 컨테이너는 재시작되어야 한다.

요구사항에서 말하는 것들

FTP

FTP란?

알아야 할 것

도커 이미지 생성

필요한 이미지

• FTPS 서버 (21)

• phpmyadmin 서버 (5000)

• nginx 서버 (80/443/22)

• 워드프레스 서버 (5050)

• Grafana 서버 (3000)

• MySQL 서버 (내부)

• InfluxDB 서버 (내부)

alpine linux와 docker

Why alpine?

docker shell 테스트 환경

$> docker pull alpine:latest
latest: Pulling from library/alpine
596ba82af5aa: Pull complete
Digest: sha256:d9a7354e3845ea8466bb00b22224d9116b183e594527fb5b6c3d30bc01a20378
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest
$> docker run -it --rm alpine:latest sh
/ #

alpine linux에서 FTP 서버 구성

$> docker run -it --rm -p 21:21 -p 42420:42420 alpine:latest sh/ #

# 익명 연결 금지
anonymous_enable=NO
# 로컬 계정으로 ftp 접속을 가능하게 함
local_enable=YES
# 사용자 홈폴더에서만 접속할 수 있게 함.
chroot_local_user=YES
seccomp_sandbox=NO
allow_writeable_chroot=YES
# 기본적인 업로드 허용
write_enable=YES
# 업로드 되는 파일 권한 설정 (생성되는 파일권한 = 777 - local_umask)
local_umask=022
# 외부에서 연결요청 받아들임
listen=YES
# 패시브 연결을 기본값으로
pasv_enable=YES
# 패시브 연결에 할당될 수 있는 아이피 최소값~최댓값 범위
pasv_min_port=42420
pasv_max_port=42420
# 패시브 연결에 사용될 아이피
pasv_address=127.0.0.1
# SSL (FTPS) 사용
ssl_enable=YES
# SSL 인증서
rsa_cert_file=/certs/cert.crt
# SSL 키
rsa_private_key_file=/certs/key.key

/ # apk --no-cache add vsftpd
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
...
OK: 7 MiB in 17 packages
/ # adduser joopark
Changing password for joopark
New password:
Bad password: similar to username
Retype password:
passwd: password for joopark changed by root
/ # mkdir certs
/ # cd certs/
/certs # vi key.key
/certs # vi cert.crt
/certs # cd ..
/ # vi vsftpd.conf
/ # vsftpd vsftpd.conf

alpine linux에서 MySQL 서버 구성

$> docker run -it --rm -p 21:21 -p 3306:3306 alpine:latest sh/ #

/ # apk --no-cache add mysql
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
...
OK: 150 MiB in 30 packages
/ # mysql_install_db --user=root --datadir=/var/lib/mysql
Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system


Two all-privilege accounts were created.
One is root@localhost, it has no password, but you need to
be system 'root' user to connect. Use, for example, sudo mysql
The second is root@localhost, it has no password either, but
you need to be the system 'root' user to connect.
After connecting you can set the password, if you would need to be
able to connect as any of these users with a password and without sudo

See the MariaDB Knowledgebase at https://mariadb.com/kb or the
MySQL manual for more instructions.

You can start the MariaDB daemon with:
cd '/usr' ; /usr/bin/mysqld_safe --datadir='/var/lib/mysql'

You can test the MariaDB daemon with mysql-test-run.pl
cd '/usr/mysql-test' ; perl mysql-test-run.pl

Please report any problems at https://mariadb.org/jira

The latest information about MariaDB is available at https://mariadb.org/.
You can find additional information about the MySQL part at:
https://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

/ # mysqld_safe --user=root --datadir=/var/lib/mysql

alpine linux에서 phpmyadmin 구성

$> docker run -it --rm -p 5000:5000 -p 3306:3306 alpine:latest sh

$> docker run -it --rm -p 5000:5000 alpine:latest sh/ #

/ # apk --no-cache add nginx php7-mysqli php7-fpmfetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gzfetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz...Executing busybox-1.32.1-r0.triggerOK: 15 MiB in 28 packages

/ # find / -name "nginx.conf"./etc/nginx/nginx.conf

# nginx.conf

http {
    server {
        root /var/www/html/page1;
        listen 80;
        index index.php;
        location / {
            try_files $uri $uri/ =404;
        }
    }
    server {
        root /var/www/html/page2;
        listen 8080;
        index index.php;
        location / {
            index index.html;
        }
    }
}

        # Includes virtual hosts configs.
        include /etc/nginx/http.d/*.conf;

server {
    listen 5000;
    root /html;
    autoindex on;
    index index.html;
    location / {
        try_files $uri $uri/ =404;
    }
}

/ # nginx
nginx: [emerg] open() "/run/nginx/nginx.pid" failed (2: No such file or directory)

/ # mkdir /run/nginx
/ # nginx

/ # nginx -g "daemon off;"

server {
    listen 5000;
    root /html;
    index index.html index.php;
    location / {
        try_files $uri $uri/ =404;
    }
    location ~ \.php$ {
        include fastcgi.conf;
        fastcgi_pass 127.0.0.1:9000;
    }
}

<?php
phpinfo();
?>

/ # php-fpm7

/ # wget https://files.phpmyadmin.net/phpMyAdmin/5.0.4/phpMyAdmin-5.0.4-all-languages.tar.gz
...
/ # tar -zxvf phpMyAdmin-5.0.4-all-languages.tar.gz --strip-components=1 -C /html
...

thrown in /html/vendor/symfony/polyfill-mbstring/Mbstring.php on line 501" while reading response header from upstream, client: 172.17.0.1, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1:5000"
2021/01/26 10:46:05 [error] 32#32: *1 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function Symfony\Polyfill\Mbstring\iconv_strpos() in /html/vendor/symfony/polyfill-mbstring/Mbstring.php:501
Stack trace:
#0 /html/vendor/symfony/polyfill-mbstring/bootstrap.php(60): Symfony\Polyfill\Mbstring\Mbstring::mb_strpos()
#1 /html/libraries/classes/Url.php(257): mb_strpos()
#2 /html/libraries/classes/Url.php(208): PhpMyAdmin\Url::getArgSeparator()
#3 /html/libraries/classes/Url.php(171): PhpMyAdmin\Url::getCommonRaw()
#4 /html/libraries/classes/Core.php(765): PhpMyAdmin\Url::getCommon()
#5 /html/libraries/classes/Core.php(338): PhpMyAdmin\Core::linkURL()
#6 /html/libraries/classes/Core.php(367): PhpMyAdmin\Core::getPHPDocLink()
#7 /html/libraries/classes/Core.php(1009): PhpMyAdmin\Core::warnMissingExtension()
#8 /html/libraries/common.inc.php(110): PhpMyAdmin\Core::checkExtensions()
#9 /html/index.php(23): require_once('/html/libraries...')
#10 {main}
  thrown in /html/vendor/symfony/polyfill-mbstring/Mbstring.php on line 501" while reading response header from upstream, client: 172.17.0.1, server: , request: "GET /index.php HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1:5000"
2021/01/26 10:59:50 [error] 60#60: *3 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function Twig\json_encode() in /html/vendor/twig/twig/src/ExtensionSet.php:116
Stack trace:
#0 /html/vendor/twig/twig/src/Environment.php(984): Twig\ExtensionSet->getSignature()
#1 /html/vendor/twig/twig/src/Environment.php(703): Twig\Environment->updateOptionsHash()
#2 /html/vendor/twig/twig/src/Environment.php(128): Twig\Environment->addExtension()
#3 /html/libraries/classes/Template.php(66): Twig\Environment->__construct()
#4 /html/libraries/classes/Core.php(299): PhpMyAdmin\Template->__construct()
#5 /html/libraries/classes/Core.php(376): PhpMyAdmin\Core::fatalError()
#6 /html/libraries/classes/Core.php(1009): PhpMyAdmin\Core::warnMissingExtension()
#7 /html/libraries/common.inc.php(110): PhpMyAdmin\Core::checkExtensions()
#8 /html/index.php(23): require_once('/html/libraries...')
#9 {main}
  thrown in /html/vendor/twig/twig/src/ExtensionSet.php on line 116" while reading response header from upstream, client: 172.17.0.1, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1:5000"

/ # apk --no-cache add php7-mbstring php7-json php7-session

create user 'joopark'@'%' identified by 'joopark';
grant all privileges on *.* to 'joopark'@'%';
flush privileges;

/ # cat test.sql | mysqld --datadir=/var/lib/mysql --user=root --bootstrap --skip-grant-tables=0

$cfg['Servers'][$i]['host'] = '172.17.0.2:3306';
$cfg['blowfish_secret'] = '{^QP+-(3mlHy+Gd~FE3mN{gIATs^1lX+T=KVYv{ubK*U0V';

wordpress 구성

$> docker run -it --rm -p 5050:5050 alpine:latest sh/ #

/ # apk --no-cache add nginx php7-mysqli php7-fpm

sudo apt install php7.3-common php7.3-mysql php7.3-curl php7.3-json php7.3-mbstring php7.3-xml php7.3-zip php7.3-gd php7.3-soap php7.3-ssh2 php7.3-tokenizer

/ # apk --no-cache add php7-common php7-curl php7-json php7-mbstring php7-xml php7-zip php7-gd php7-soap php7-ssh2 php7-tokenizer

/ # wget https://ko.wordpress.org/latest-ko_KR.tar.gz
/ # mkdir html
/ # tar -zxvf latest-ko_KR.tar.gz --strip-components=1 -C /html
/ # mkdir /run/nginx
/ # vi /etc/nginx/http.d/wordpress.conf
/ # php-fpm7
/ # cp /html/wp-config-sample.php /html/wp-config.php
/ # dos2unix /html/wp-config.php
/ # vi /html/wp-config.php
/ # nginx -g "daemon off;"

Grafana / InfluxDB

https://medium.com/naver-cloud-platform/grafana-influxdb를-활용한-모니터링-서비스-구축하기-62de4b07a505
https://musma.github.io/2019/07/08/getting-started-with-influxdb-time-series-database.html
https://www.popit.kr/influxdb_telegraf_grafana_1/

Grafana

InfluxDB

InfluxDB 설치

$> docker run -it --rm -p 8083:8083 -p 8086:8086 alpine:latest sh/ #

/ # apk --no-cache add influxdb

/ # influx
influx          influx_inspect  influx_stress   influx_tools    influx_tsm      influxd

/ # influxd -h
Configure and start an InfluxDB server.

Usage: influxd [[command] [arguments]]

The commands are:

    backup               downloads a snapshot of a data node and saves it to disk
    config               display the default configuration
    help                 display this help message
    restore              uses a snapshot of a data node to rebuild a cluster
    run                  run node with existing configuration
    version              displays the InfluxDB version

"run" is the default command.

Use "influxd [command] -help" for more information about a command.
/ # influxd run -help
Runs the InfluxDB server.

Usage: influxd run [flags]

    -config <path>
            Set the path to the configuration file.
            This defaults to the environment variable INFLUXDB_CONFIG_PATH,
            ~/.influxdb/influxdb.conf, or /etc/influxdb/influxdb.conf if a file
            is present at any of these locations.
            Disable the automatic loading of a configuration file using
            the null device (such as /dev/null).
    -pidfile <path>
            Write process ID to a file.
    -cpuprofile <path>
            Write CPU profiling information to a file.
    -memprofile <path>
            Write memory usage information to a file.

run: flag: help requested

/ # find . -name "influxdb.conf"
./etc/influxdb.conf
/ # influxd run -config /etc/influxdb.conf

 8888888           .d888 888                   8888888b.  888888b.
   888            d88P"  888                   888  "Y88b 888  "88b
   888            888    888                   888    888 888  .88P
   888   88888b.  888888 888 888  888 888  888 888    888 8888888K.
   888   888 "88b 888    888 888  888  Y8bd8P' 888    888 888  "Y88b
   888   888  888 888    888 888  888   X88K   888    888 888    888
   888   888  888 888    888 Y88b 888 .d8""8b. 888  .d88P 888   d88P
 8888888 888  888 888    888  "Y88888 888  888 8888888P"  8888888P"

2021-01-27T14:03:05.280643Z info    InfluxDB starting
... 중략 ...

$> docker ps
CONTAINER ID   IMAGE           COMMAND   CREATED          STATUS          PORTS                                            NAMES
f9e30cc6e6a0   alpine:latest   "sh"      13 minutes ago   Up 13 minutes   0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp   recursing_burnell
$> docker exec -it f9e3 sh
/ # influx
Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
$>

$> curl -i -XPOST 'http://127.0.0.1:8086/query' --data-binary "q=CREATE DATABASE people"
$> curl -i -XPOST 'http://127.0.0.1:8086/write?db=people' --data-binary 'pujuhong,region=korea temp=36,blood_pressure_min=80,blood_pressure_max=120'

pujuhong,region=korea temp=36,blood_pressure_min=80,blood_pressure_max=120

[measurement],[태그명]=[태그값][공백][필드명]=[필드값]

> show databases
name: databases
name
----
_internal
people
> use people
Using database people
> show measurements
name: measurements
name
----
pujuhong
> select * from pujuhong
name: pujuhong
time                blood_pressure_max blood_pressure_min region temp
----                ------------------ ------------------ ------ ----
1611762474305246900 120                80                 korea  36
>

telegraf

/ # apk --no-cache add telegraf

/ # telegraf -sample-config -input-filter cpu:disk:kernel:mem:net:netstat:system:mysql -output-filter influxdb > telegraf.conf

[[outputs.influxdb]]
...
 urls = ["http://[Influx 컨테이너 아이피]:8086"]
...
 database = "mysql"

/ # telegraf
/ # telegraf --help
Telegraf, The plugin-driven server agent for collecting and reporting metrics.

Usage:

  telegraf [commands|flags]

The commands & flags are:

  config              print out full sample configuration to stdout
  version             print the version to stdout

  --aggregator-filter <filter>   filter the aggregators to enable, separator is :
  --config <file>                configuration file to load
  --config-directory <directory> directory containing additional *.conf files
  --plugin-directory             directory containing *.so files, this directory will be
                                 searched recursively. Any Plugin found will be loaded
                                 and namespaced.
  --debug                        turn on debug logging
  --input-filter <filter>        filter the inputs to enable, separator is :
  --input-list                   print available input plugins.
  --output-filter <filter>       filter the outputs to enable, separator is :
  --output-list                  print available output plugins.
  --pidfile <file>               file to write our pid to
  --pprof-addr <address>         pprof address to listen on, don't activate pprof if empty
  --processor-filter <filter>    filter the processors to enable, separator is :
  --quiet                        run in quiet mode
  --section-filter               filter config sections to output, separator is :
                                 Valid values are 'agent', 'global_tags', 'outputs',
                                 'processors', 'aggregators' and 'inputs'
  --sample-config                print out full sample configuration
  --once                         enable once mode: gather metrics once, write them, and exit
  --test                         enable test mode: gather metrics once and print them
  --test-wait                    wait up to this many seconds for service
                                 inputs to complete in test or once mode
  --usage <plugin>               print usage for a plugin, ie, 'telegraf --usage mysql'
  --version                      display the version and exit

Examples:

  # generate a telegraf config file:
  telegraf config > telegraf.conf

  # generate config with only cpu input & influxdb output plugins defined
  telegraf --input-filter cpu --output-filter influxdb config

  # run a single telegraf collection, outputting metrics to stdout
  telegraf --config telegraf.conf --test

  # run telegraf with all plugins defined in config file
  telegraf --config telegraf.conf

  # run telegraf, enabling the cpu & memory input, and influxdb output plugins
  telegraf --config telegraf.conf --input-filter cpu:mem --output-filter influxdb

  # run telegraf with pprof
  telegraf --config telegraf.conf --pprof-addr localhost:6060
/ # telegraf --config telegraf.conf
2021-01-28T02:40:38Z I! Starting Telegraf 1.17.0
2021-01-28T02:40:38Z I! Loaded inputs: cpu disk kernel mem mysql net netstat system
2021-01-28T02:40:38Z I! Loaded aggregators:
2021-01-28T02:40:38Z I! Loaded processors:
2021-01-28T02:40:38Z I! Loaded outputs: influxdb
2021-01-28T02:40:38Z I! Tags enabled: host=adf2d78c04f7
2021-01-28T02:40:38Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"adf2d78c04f7", Flush Interval:10s

> show databases
name: databases
name
----
_internal
people
mysql
> use mysql
Using database mysql
> show measurements
name: measurements
name
----
cpu
disk
kernel
mem
mysql
net
netstat
system
> select * from kernel
name: kernel
time                boot_time  context_switches entropy_avail host         interrupts processes_forked
----                ---------  ---------------- ------------- ----         ---------- ----------------
1611801640000000000 1611397030 395734441        3185          adf2d78c04f7 231201864  229577
1611801650000000000 1611397030 395741523        3187          adf2d78c04f7 231206364  229580
1611801660000000000 1611397030 395750344        3190          adf2d78c04f7 231211699  229588
1611801670000000000 1611397030 395760634        3192          adf2d78c04f7 231217933  229595
1611801680000000000 1611397030 395770884        3195          adf2d78c04f7 231223879  229596
1611801690000000000 1611397030 395782705        3197          adf2d78c04f7 231230325  229596
...

Grafana 설치 및 실행

$> docker run -it --rm --name grafana -p 3000:3000 alpine:latest sh
/ #

/ # apk --no-cache add grafana
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/1) Installing grafana (7.3.6-r0)
Executing grafana-7.3.6-r0.pre-install
Executing busybox-1.32.1-r0.trigger
OK: 171 MiB in 15 packages
/ # grafana-
grafana-cli     grafana-server
/ # grafana-server --help
Usage of grafana-server:
  -config string
    	path to config file
  -convey-json
    	When true, emits results in JSON blocks. Default: 'false'
  -convey-silent
    	When true, all output from GoConvey is suppressed.
  -convey-story
    	When true, emits story output, otherwise emits dot output. When not provided, this flag mirrors the value of the '-test.v' flag
  -homepath string
    	path to grafana install/home path, defaults to working directory
  -packaging string
    	describes the way Grafana was installed (default "unknown")
  -pidfile string
    	path to pid file
  -profile
    	Turn on pprof profiling
  -profile-port uint
    	Define custom port for profiling (default 6060)
  -tracing
    	Turn on tracing
  -tracing-file string
    	Define tracing output file (default "trace.out")
  -v	prints current version and exits
/ # grafana-server
Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath
/ #

/ # find . -name "defaults.ini"
./usr/share/grafana/conf/defaults.ini
/ # grafana-server -config /usr/share/grafana/conf/defaults.ini
Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath

/ # grafana-server -homepath /usr/share/grafana
INFO[01-28|03:16:34] Starting Grafana                         logger=server version=7.3.6 commit=f25d63954b branch=master compiled=2020-12-17T09:54:28+0000
INFO[01-28|03:16:34] Config loaded from                       logger=settings file=/usr/share/grafana/conf/defaults.ini
INFO[01-28|03:16:34] Path Home                                logger=settings path=/usr/share/grafana
INFO[01-28|03:16:34] Path Data                                logger=settings path=/usr/share/grafana/data
INFO[01-28|03:16:34] Path Logs                                logger=settings path=/usr/share/grafana/data/log
INFO[01-28|03:16:34] Path Plugins                             logger=settings path=/usr/share/grafana/data/plugins
INFO[01-28|03:16:34] Path Provisioning                        logger=settings path=/usr/share/grafana/conf/provisioning
...

nginx 서버

• 포트 80으로 들어오는 요청은 443으로 리다이렉션

• /wordpress 로 접근할 시 307 리다이렉트

• /phpmyadmin 으로 접근할 시 리버스 프록시

• ssh로 접근가능

$> docker run -it --rm -p 80:80 -p 443:443 -p 22:22 -p 5000:5000 -p 5050:5050 alpine:latest sh
/ # apk --no-cache add nginx openssh
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
...
OK: 13 MiB in 25 packages
/ #

/ # sshd
sshd re-exec requires execution with an absolute path
/ # find . -name "sshd"
./usr/sbin/sshd
./etc/conf.d/sshd
./etc/init.d/sshd
/ # /usr/sbin/sshd
sshd: no hostkeys available -- exiting.

/ # ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ''
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
The key fingerprint is:
SHA256:Wtnb9iyniMN4v8cXwNNmHhHLDnNfuArhgrUr/riy3Ok root@98b40ef287fd
The key's randomart image is:
+---[RSA 3072]----+
|              .. |
|             ..o |
|        . ..o.=..|
|       o = .+==o.|
|      . S +  *o..|
|       o o + .o  |
|      oo. ..+  . |
|   ..o.++. o+oo  |
|    o+E+oo+o.=o  |
+----[SHA256]-----+

• t : 만들 암호쌍의 알고리즘을 정함

• f : 저장할 암호쌍의 경로를 정함

• N : 암호쌍에 대한 비밀번호를 입력하게 되어 있는데 비밀번호는 사용하지 않는다.

/ # /usr/sbin/sshd
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh
   23 root      0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
   24 root      0:00 ps
/ #

$> ssh root@127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is SHA256:Wtnb9iyniMN4v8cXwNNmHhHLDnNfuArhgrUr/riy3Ok.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts.
root@127.0.0.1's password:
Permission denied, please try again.
root@127.0.0.1's password:
Permission denied, please try again.
root@127.0.0.1's password:
root@127.0.0.1: Permission denied (publickey,password,keyboard-interactive).

• passwd

• chpasswd

/ # vi /etc/ssh/sshd_config
/ # echo 'root:joopark' | chpasswd
chpasswd: password for 'root' changed

/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh
   23 root      0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
   33 root      0:00 ps
/ # kill 23
/ # /usr/sbin/sshd

$> ssh root@127.0.0.1
root@127.0.0.1's password:
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

98b40ef287fd:~#

/ # mkdir cert
/ # mkdir html
/ # vi /cert/cert.crt
/ # vi /cert/key.key
/ # vi /html/index.html
/ # rm /etc/nginx/http.d/default.conf
/ # vi /etc/nginx/http.d/http.conf
/ # vi /etc/nginx/http.d/https.conf

server {
    listen 80;
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    ssl_certificate /cert/cert.crt;
    ssl_certificate_key /cert/key.key;
    ssl_session_timeout 5m;
    ssl_prefer_server_ciphers on;
    index index.html index.htm;
    root /html;

    location / {
        try_files $uri $uri/ =404;
    }

    location /wordpress {
        return 307 http://172.17.0.3:5050;
    }

    location /phpmyadmin {
        proxy_pass http://172.17.0.3:5000;
    }
}

다시 쿠버네티스로

minikube

$> minikube start
😄  Darwin 11.1 위의 minikube v1.17.0
✨  자동적으로 docker 드라이버가 선택되었습니다. 다른 드라이버 목록: hyperkit, ssh
👍  minikube 클러스터의 minikube 컨트롤 플레인 노드를 시작하는 중
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v8-v1....: 491.22 MiB / 491.22 MiB  100.00% 35.35 Mi
🔥  Creating docker container (CPUs=2, Memory=1988MB) ...
🐳  쿠버네티스 v1.20.2 을 Docker 20.10.2 런타임으로 설치하는 중
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

kubectl

kubectl get

$> kubectl get pod
$> kubectl get replicaset
$> kubectl get node
$> kubectl get service
...

kubecel delete

$> kubectl delete pod/[name]
$> kubectl delete replicaset/[name]
$> kubectl delete node/[name]
$> kubectl delete service/[name]
...

pod 만들어 보기

FROM alpine:latest

COPY query.sql /tmp/init/
COPY mariadb-server.cnf /tmp/init/

RUN apk --no-cache add mysql \
&& mysql_install_db --user=root --datadir=/var/lib/mysql \
&& mv /tmp/init/mariadb-server.cnf /etc/my.cnf.d/ \
&& cat /tmp/init/query.sql | mysqld --datadir=/var/lib/mysql --user=root --bootstrap --skip-grant-tables=0

EXPOSE 3306

CMD ["mysqld_safe", "--user=root", "--datadir=/var/lib/mysql"]

$> docker build -t "ft_services/mysql:0.1" .

$> kubectl run mysql --image=ft_services/mysql:0.1
pod/mysql created
$> kubectl get pod
NAME    READY   STATUS             RESTARTS   AGE
mysql   0/1     ImagePullBackOff   0          56s

$> kubectl delete pod/mysql
pod "mysql" deleted
$> eval $(minikube docker-env)
$> docker build -t "ft_services/mysql:0.1" [dockerfile 경로]
...중략...
Successfully tagged ft_services/mysql:0.1
$> kubectl run mysql --image=ft_services/mysql:0.1
pod/mysql created
$> kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
mysql   1/1     Running   0          3s

apiVersion: v1
kind: Pod # 생성하고자 하는 자원의 종류 (pod 이외에도 레플리카셋 서비스 등이 올수있음)
metadata: # 생성하고자 하는 자원의 메타데이터 (이름, 라벨 등)
  name: mysql # 이름
  labels:
    app: mysql
spec: # 상세한 자원 명세
  containers:
    - name: app
      image: ft_services/mysql:0.1

$> kubectl apply -f mysql_pod.yml
pod/mysql created
$> kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
mysql   1/1     Running   0          3s
$> kubectl delete -f mysql_pod.yml
pod "mysql" deleted

apiVersion: v1
kind: Pod # 생성하고자 하는 자원의 종류 (pod 이외에도 레플리카셋 서비스 등이 올수있음)
metadata: # 생성하고자 하는 자원의 메타데이터 (이름, 라벨 등)
  name: mysql # 이름
  labels:
    app: mysql
spec: # 상세한 자원 명세
  containers:
    - name: app
      image: ft_services/mysql:0.1
      livenessProbe:
        tcpSocket: # TCP 연결을 진행해보아 해당 포트가 살아있는지 확인한다.
          port: 3306
        initialDelaySeconds: 5 # 5초동안 기다렸다가 검사함
        timeoutSeconds: 2 # Default 1
        periodSeconds: 5 # Defaults 10
        failureThreshold: 1 # Defaults 3

레플리카셋

apiVersion: apps/v1
kind: ReplicaSet # 생성하고자 하는 자원의 종류 (레플리카셋)
metadata:
  name: mysqlp
spec:
  replicas: 3 # 복제본 개수
  selector: # 레플리카셋으로 구분되는 조건
    matchLabels:
      app: mysql # 라벨1
      tier: app # 라벨2
  template: # 생성할 pod들의 템플릿
    metadata:
      labels:
        app: mysql
        tier: app
    spec:
      containers:
        - name: mysql
          image: ft_services/mysql:0.1

$> kubectl apply -f mysql_rs.yml
replicaset.apps/mysqlp created
$> kubectl get replicaset
NAME     DESIRED   CURRENT   READY   AGE
mysqlp   3         3         3       68s
$> kubectl get pod
NAME           READY   STATUS    RESTARTS   AGE
mysqlp-lvwvp   1/1     Running   0          70s
mysqlp-prrwd   1/1     Running   0          70s
mysqlp-znp7q   1/1     Running   0          70s
$> kubectl delete pod mysqlp-lvwvp
pod "mysqlp-lvwvp" deleted
$> kubectl get pod
NAME           READY   STATUS    RESTARTS   AGE
mysqlp-5g4qp   1/1     Running   0          2m17s
mysqlp-prrwd   1/1     Running   0          11m
mysqlp-znp7q   1/1     Running   0          11m
$> kubectl delete -f mysql_rs.yml
replicaset.apps "mysqlp" deleted

디플로이먼트

apiVersion: apps/v1
kind: Deployment # 생성하고자 하는 자원의 종류 (디플로이먼트)
metadata:
  name: mysqlp
spec:
  replicas: 3 # 복제본 개수
  selector: # 레플리카셋으로 구분되는 조건
    matchLabels:
      app: mysql # 라벨1
      tier: app # 라벨2
  template: # 생성할 pod들의 템플릿
    metadata:
      labels:
        app: mysql
        tier: app
    spec:
      containers:
        - name: mysql
          image: ft_services/mysql:0.1

$> kubectl apply -f mysql_deploy.yml
deployment.apps/mysqlp created
$> kubectl get deployment
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
mysqlp   3/3     3            3           12s
$> kubectl get replicaset
NAME               DESIRED   CURRENT   READY   AGE
mysqlp-8ff58c8d8   3         3         3       8s
$> kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
mysqlp-8ff58c8d8-d9pfj   1/1     Running   0          9m44s
mysqlp-8ff58c8d8-snd58   1/1     Running   0          9m43s
mysqlp-8ff58c8d8-zknl2   1/1     Running   0          9m43s
$> kubectl delete pod mysqlp-8ff58c8d8-d9pfj
pod "mysqlp-8ff58c8d8-d9pfj" deleted
$> kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
mysqlp-8ff58c8d8-jr8vk   1/1     Running   0          4m21s
mysqlp-8ff58c8d8-snd58   1/1     Running   0          14m
mysqlp-8ff58c8d8-zknl2   1/1     Running   0          14m
$> kubectl delete -f mysql_deploy.yml
deployment.apps "mysqlp" deleted

Service

ClusterIP

FROM alpine:latest

COPY wordpress.conf /tmp/init/
COPY wp-config.php /tmp/init/

RUN apk --no-cache add nginx php7-mysqli php7-fpm php7-common php7-curl php7-json php7-mbstring php7-xml php7-zip php7-gd php7-soap php7-ssh2 php7-tokenizer \
&& wget https://ko.wordpress.org/latest-ko_KR.tar.gz \
&& mkdir html \
&& mkdir /run/nginx \
&& tar -zxvf latest-ko_KR.tar.gz --strip-components=1 -C /html \
&& mv /tmp/init/wordpress.conf /etc/nginx/http.d \
&& mv /tmp/init/wp-config.php /html \
&& dos2unix /html/wp-config.php \
&& php-fpm7

EXPOSE 5050
EXPOSE 3306

ENTRYPOINT \
php-fpm7 \
&& nginx -g "daemon off;"

apiVersion: v1
kind: Service
metadata:
  name: mysql-pod # 이 이름으로 다른 컨테이너 (파드)가 하기에 명세하는 컨테이너로 접속 가능
spec:
  ports:
    - port: 3306
      protocol: TCP
  selector: # 이 설정을 가진 파드에 접근할 수 있다.
    app: mysql
    tier: app

NodePort

apiVersion: v1
kind: Service
metadata:
  name: nodeport-wordpress # 노드포트 서비스 이름
spec:
  type: NodePort # 타입을 노드포트로 정의
  ports:
    - port: 5050 # 클러스터 내부의 포트
      protocol: TCP
      nodePort: 31000 # 클러스터 외부의 포트 (30000-32768 까지 할당 가능)
  selector: # 연결할 대상
    app: ft_services
    tier: wordpress

$> minikube ip
192.168.64.6

LoadBalancer

apiVersion: v1
kind: Service
metadata:
  name: nodeport-wordpress # 노드포트 서비스 이름
spec:
  type: LoadBalancer # 타입을 노드포트로 정의
  ports:
    - targetPort: 5050 # 클러스터 내부의 포트
      protocol: TCP
      port: 31000 # 클러스터 외부의 포트 (30000-32768 까지 할당 가능)
  selector: # 연결할 대상
    app: ft_services
    tier: wordpress

컴퓨터공학에서 Bare Metal의 본래 뜻은 소프트웨어가 설치되지 않은 하드웨어를 의미한다. 따라서 Bare Metal은 여러 가지 의미로 쓰일 수 있는데 사용자가 빈 컴퓨터 내에 OS를 직접 설치해서 사용하는 것을 베어메탈이라 할 수도 있으며 베어메탈을 가상화에 응용할 수도 있다. (기존 가상머신은 OS 레벨로 가상화를 하지만 베어메탈 가상화는 가상머신이 하드웨어(베어메탈)에 직접 접근할 수 있도록 한다)

일반적으로 쿠버네티스는 IaaS (간단하게 말하면 아마존 AWS나 구글 GCP 등) 위에서 구동되며 그런 IaaS가 로드밸런서를 별도로 제공해 준다. 하지만 현재 로컬에 설치하는 쿠버네티스에는 당연히 로드밸런서가 기본적으로 존재하지 않으므로 쿠버네티스에서 로드밸런스 사양을 명시해도 보류 상태로 유지될 수밖에 없다.

아마도 쿠버네티스를 개인 직접 설치해서 사용하면 일반적으로 IaaS같은 네트워크 환경이 구성되어 있지 않으므로 이러한 것을 도와주는 프로그램 인 것 같다.

$> minikube addons list
|-----------------------------|----------|--------------|
|         ADDON NAME          | PROFILE  |    STATUS    |
|-----------------------------|----------|--------------|
| ambassador                  | minikube | disabled     |
| csi-hostpath-driver         | minikube | disabled     |
| dashboard                   | minikube | disabled     |
| default-storageclass        | minikube | enabled ✅   |
| efk                         | minikube | disabled     |
| freshpod                    | minikube | disabled     |
| gcp-auth                    | minikube | disabled     |
| gvisor                      | minikube | disabled     |
| helm-tiller                 | minikube | disabled     |
| ingress                     | minikube | disabled     |
| ingress-dns                 | minikube | disabled     |
| istio                       | minikube | disabled     |
| istio-provisioner           | minikube | disabled     |
| kubevirt                    | minikube | disabled     |
| logviewer                   | minikube | disabled     |
| metallb                     | minikube | disabled     |
| metrics-server              | minikube | disabled     |
| nvidia-driver-installer     | minikube | disabled     |
| nvidia-gpu-device-plugin    | minikube | disabled     |
| olm                         | minikube | disabled     |
| pod-security-policy         | minikube | disabled     |
| registry                    | minikube | disabled     |
| registry-aliases            | minikube | disabled     |
| registry-creds              | minikube | disabled     |
| storage-provisioner         | minikube | enabled ✅   |
| storage-provisioner-gluster | minikube | disabled     |
| volumesnapshots             | minikube | disabled     |
|-----------------------------|----------|--------------|

$> minikube addons enable metallb
🌟  'metallb' 애드온이 활성화되었습니다

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system # MetalLB
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses: # CIDR 혹은 아이피 대역으로 표시함.
      - 192.168.64.6-192.168.64.6 # minikube ip

yaml 파일로 클러스터 생성해 놓기

• 디플로이먼트
• WordPress : wordpress 동작
• Nginx : https 웹서버, ssh 및 프록시, 리다이렉트
• PhpMyAdmin : phpmyadmin 동작
• FTPS : 파일 서버
• MySQL : DB
• Grafana : 수집한 데이터 통계
• InfluxDB : 실행되는 노드들의 데이터 수집

• 로드밸런서
• grafana : 3000
• wordpress : 5050
• nginx : 80/443/22
• phpmyadmin : 5000
• ftps : 21 (외에 포트 하나 더 필요함)

• 클러스터 IP
• nginx -> phpmyadmin : 5000
• 다른 노드들 -> InFluxDB : 8086
• phpmyadmin, wordpress -> MySQL : 3306

Deployment

WordPress

apiVersion: apps/v1
kind: Deployment # 생성하고자 하는 자원의 종류 (디플로이먼트)
metadata:
  name: wordpress # Deployment Name
spec:
  replicas: 1 # pod 복제본 개수
  selector: # 구분되는 조건
    matchLabels:
      app: ft_services # 라벨1 (앱 이름)
      tier: wordpress # 라벨2
  template: # 생성할 pod들의 템플릿
    metadata:
      labels:
        app: ft_services # 라벨1 (앱 이름)
        tier: wordpress # 라벨2
    spec:
      containers:
        - name: wordpress # 컨테이너 이름
          image: ft_services/wordpress:0.1

Nginx ~ InfluxDB

클러스터 IP

phpmyadmin

apiVersion: v1
kind: Service
metadata:
  name: phpmyadmin-ip
spec:
  ports:
    - port: 5000
      protocol: TCP
  selector: # 이 설정을 가진 파드에 접근할 수 있다.
    app: ft_services
    tier: phpmyadmin

InfluxDB

apiVersion: v1
kind: Service
metadata:
  name: influxdb-ip
spec:
  ports:
    - port: 8086
      protocol: TCP
  selector: # 이 설정을 가진 파드에 접근할 수 있다.
    app: ft_services
    tier: influxdb

MySQL

apiVersion: v1
kind: Service
metadata:
  name: mysql-ip
spec:
  ports:
    - port: 3306
      protocol: TCP
  selector: # 이 설정을 가진 파드에 접근할 수 있다.
    app: ft_services
    tier: mysql

LoadBalancer

Grafana

apiVersion: v1
kind: Service
metadata:
  name: loadbalancer-grafana
spec:
  type: LoadBalancer
  ports:
    - port: 3000
      protocol: TCP
      nodePort: 33000
  selector:
    app: ft_services
    tier: wordpress

Wordpress ~ FTPS

$> kubectl get deployment
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
ftps         1/1     1            1           9h
grafana      1/1     1            1           149m
influxdb     1/1     1            1           144m
mysql        1/1     1            1           3h2m
nginx        1/1     1            1           10h
phpmyadmin   1/1     1            1           10h
wordpress    1/1     1            1           21h
$> kubectl get services
NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                   AGE
influxdb-ip               ClusterIP      10.99.66.176     <none>         8086/TCP                                  18s
kubernetes                ClusterIP      10.96.0.1        <none>         443/TCP                                   24h
loadbalancer-ftps         LoadBalancer   10.110.115.37    192.168.64.7   21:31292/TCP,42420:30282/TCP              98m
loadbalancer-grafana      LoadBalancer   10.110.135.139   <pending>      3000:32614/TCP                            98m
loadbalancer-nginx        LoadBalancer   10.106.156.226   <pending>      80:31572/TCP,443:30505/TCP,22:31255/TCP   98m
loadbalancer-phpmyadmin   LoadBalancer   10.111.241.186   <pending>      5000:30570/TCP                            98m
loadbalancer-wordpress    LoadBalancer   10.98.156.141    <pending>      5050:30406/TCP                            98m
mysql-ip                  ClusterIP      10.102.97.225    <none>         3306/TCP                                  18s
phpmyadmin-ip             ClusterIP      10.111.183.158   <none>         5000/TCP                                  18s
$>

한 로드밸런서 외에 다른 로드밸런서에 아이피가 할당이 안되는 문제

metadata:
  annotations:
    metallb.universe.tf/allow-shared-ip: shared

$> kubectl get services
NAME                      TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                   AGE
influxdb-ip               ClusterIP      10.99.66.176     <none>         8086/TCP                                  12m
kubernetes                ClusterIP      10.96.0.1        <none>         443/TCP                                   24h
loadbalancer-ftps         LoadBalancer   10.99.99.39      192.168.64.7   21:31944/TCP,42420:31317/TCP              4s
loadbalancer-grafana      LoadBalancer   10.108.126.108   192.168.64.7   3000:32145/TCP                            4s
loadbalancer-nginx        LoadBalancer   10.110.119.144   192.168.64.7   80:32284/TCP,443:31625/TCP,22:31430/TCP   4s
loadbalancer-phpmyadmin   LoadBalancer   10.109.37.175    192.168.64.7   5000:32099/TCP                            3s
loadbalancer-wordpress    LoadBalancer   10.101.71.77     192.168.64.7   5050:31458/TCP                            3s
mysql-ip                  ClusterIP      10.102.97.225    <none>         3306/TCP                                  12m
phpmyadmin-ip             ClusterIP      10.111.183.158   <none>         5000/TCP                                  12m
$>

FTPS 동작 확인

$> kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
ftps-6d946fdd7f-75zxg         1/1     Running   2          22h
grafana-6dbc757b84-j55n2      1/1     Running   0          15h
influxdb-85b6c8bcdb-qsjtn     1/1     Running   0          15h
mysql-6598db6b58-zf5p5        1/1     Running   0          16h
nginx-bdb46589b-7mv6z         1/1     Running   0          23h
phpmyadmin-7c7c6bdfc7-z2h6k   1/1     Running   0          22h
wordpress-57894bcd45-9r5mj    1/1     Running   0          34h
$> kubectl exec -it ftps-6d946fdd7f-75zxg -- sh
/ #

        - name: ftps # 컨테이너 이름
          image: ft_services/ftps:0.1
          command: ["sleep"]
          args: ["inf"]

$> kubectl apply -f ftps/ftps_deploy.yml
deployment.apps/ftps configured
$> kubectl get pods
NAME                          READY   STATUS        RESTARTS   AGE
ftps-75f885d4c4-2ljjk         1/1     Terminating   2          47m
ftps-7b5447bd7f-sm58f         1/1     Running       0          13s
grafana-6dbc757b84-j55n2      1/1     Running       0          18h
influxdb-85b6c8bcdb-qsjtn     1/1     Running       0          18h
mysql-6598db6b58-zf5p5        1/1     Running       0          18h
nginx-bdb46589b-7mv6z         1/1     Running       0          26h
phpmyadmin-7c7c6bdfc7-z2h6k   1/1     Running       0          25h
wordpress-57894bcd45-9r5mj    1/1     Running       0          37h
$> kubectl exec -it ftps-7b5447bd7f-sm58f -- sh
/ # ps
PID   USER     TIME  COMMAND
    1 root      0:00 sleep inf
    9 root      0:00 sh
   15 root      0:00 ps
/ #

$> minikube ip
192.168.64.9
$> kubectl exec -it ftps-7b5447bd7f-sm58f -- sh
/ # vi vsftpd.conf
/ # vsftpd vsftpd.conf

$> sed "s/CLUSTER_IP_ADDR/$(minikube ip)/" vsftpd.conf > vsftpd.conf.tmp$> mv vsftpd.conf.tmp vsftpd.conf

Replace all occurances of `foo' with `bar' in the file test.txt, without creating a backup of the file:

sed -i '' -e 's/foo/bar/g' test.txt

$> sed -i '' -e "s/CLUSTER_IP_ADDR/$(minikube ip)/g" vsftpd.conf

phpmyadmin 동작 확인

$> kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
ftps-7b5447bd7f-84628         1/1     Running   0          158m
grafana-6dbc757b84-2v6vk      1/1     Running   0          158m
influxdb-85b6c8bcdb-tkhfl     1/1     Running   0          158m
mysql-6598db6b58-l4ftc        1/1     Running   0          158m
nginx-bdb46589b-7c6bp         1/1     Running   0          158m
phpmyadmin-7c7c6bdfc7-7rh6k   1/1     Running   0          158m
wordpress-57894bcd45-mqxd8    1/1     Running   0          158m
$> kubectl exec -it wordpress-57894bcd45-mqxd8 -- sh
/ # vi html/wp-config.php

Wordpress 테스트

$> ssh root@192.168.64.9
The authenticity of host '192.168.64.9 (192.168.64.9)' can't be established.
RSA key fingerprint is SHA256:+M0zWj6YDwpAqOOW3wfsjp/2j9J61vbLvGrLYbqUwuw.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.64.9' (RSA) to the list of known hosts.
root@192.168.64.9's password:
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

nginx-bdb46589b-7c6bp:~#

nginx 테스트

$> ssh root@192.168.64.9
The authenticity of host '192.168.64.9 (192.168.64.9)' can't be established.
RSA key fingerprint is SHA256:+M0zWj6YDwpAqOOW3wfsjp/2j9J61vbLvGrLYbqUwuw.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.64.9' (RSA) to the list of known hosts.
root@192.168.64.9's password:
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

nginx-bdb46589b-7c6bp:~#

    location /phpmyadmin/ {
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass http://phpmyadmin-ip:5000/phpmyadmin/;
    }

MySQL 설정

추가로 알아야 할 것들

ConfigMap

apiVersion: v1
kind: ConfigMap # ConfigMap
metadata:
  name: configmap-example # ConfigMap
data:
  username: "pujuhong" # 키 : 값 대응
  config.conf: | # 파일처럼 쓰는 키 :
    #!/bin/sh
    mysqld_safe --user=root --datadir=/var/lib/mysql &
    sleep 5
    mysql -u root < /tmp/init/wp_db.sql
    sleep inf

Volume

Persistent Volume

요구사항 다시 확인

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

apiVersion: apps/v1
kind: Deployment # 생성하고자 하는 자원의 종류 (디플로이먼트)
metadata:
  name: mysql # Deployment Name
spec:
  replicas: 1 # pod 복제본 개수
  selector: # 구분되는 조건
    matchLabels:
      app: ft_services # 라벨1 (앱 이름)
      tier: mysql # 라벨2
  template: # 생성할 pod들의 템플릿
    metadata:
      labels:
        app: ft_services # 라벨1 (앱 이름)
        tier: mysql # 라벨2
    spec:
      containers:
        - name: mysql # 컨테이너 이름
          image: ft_services/mysql:0.3
          volumeMounts: # 퍼시스턴스 볼륨 접근
          - name: mysql-data-v
            mountPath: /var/lib/mysql # 경로
      volumes: # 컨테이너 이름
        - name: mysql-data-v # 볼륨 이름
          persistentVolumeClaim:
            claimName: mysql-data # 클레임 네임

#!/bin/sh
if [ -d "/var/lib/mysql/mysql" ]; then
    mysqld_safe --user=root --datadir=/var/lib/mysql
else
    mysql_install_db --user=root --datadir=/var/lib/mysql
    mysqld_safe --user=root --datadir=/var/lib/mysql &
    sleep 5
    mysql -u root < /tmp/init/query.sql
    mysql -u root < /tmp/init/wp_db.sql
    sleep inf
fi

$> kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
ftps-6d946fdd7f-fch4q         1/1     Running   0          56m
grafana-6dbc757b84-lhtwj      1/1     Running   0          56m
influxdb-85b6c8bcdb-grmzm     1/1     Running   0          56m
mysql-5867bf4887-s7z75        1/1     Running   0          56m
nginx-647595897-ll746         1/1     Running   0          56m
phpmyadmin-7f8968cdf5-g952r   1/1     Running   0          56m
wordpress-57894bcd45-9q6mn    1/1     Running   0          56m
$> kubectl delete pod mysql-5867bf4887-s7z75
pod "mysql-5867bf4887-s7z75" deleted
$> kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
ftps-6d946fdd7f-fch4q         1/1     Running   0          57m
grafana-6dbc757b84-lhtwj      1/1     Running   0          57m
influxdb-85b6c8bcdb-grmzm     1/1     Running   0          57m
mysql-5867bf4887-22vqt        1/1     Running   0          44s
nginx-647595897-ll746         1/1     Running   0          57m
phpmyadmin-7f8968cdf5-g952r   1/1     Running   0          57m
wordpress-57894bcd45-9q6mn    1/1     Running   0          57m
$>

#!/bin/bash
IP=$(minikube ip)
sed -i '' -e "s/CLUSTER_IP_ADDR/$IP/g" srcs/mysql/files/wp_db.sql
# kubectl apply ...

InfluxDB

[agent]
  ## 데이터 간격 수집 인터벌
  interval = "10s"
  ## 수집하는 시간 단위를 인터벌에 맞추는가
  round_interval = true

# db data
[[outputs.influxdb]]
  ## InfluxDB URI
  urls = ["http://influxdb-ip:8086"]
  ## db name
  database = "telegraf-beta1"

# Read metrics about cpu usage
[[inputs.cpu]]
  ## cpu 사용량 수집
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics.
  collect_cpu_time = false
  ## If true, compute and report the sum of all non-idle CPU states.
  report_active = false

# Read metrics about disk usage by mount point
[[inputs.disk]]
  ## By default stats will be gathered for all mount points.
  ## Set mount_points will restrict the stats to only the specified mount points.
  # mount_points = ["/"]

  ## Ignore mount points by filesystem type.
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

# Get kernel statistics from /proc/stat
[[inputs.kernel]]
  # no configuration

# Read metrics about memory usage
[[inputs.mem]]
  # no configuration

# Read metrics about network interface usage
[[inputs.net]]
  interfaces = ["eth0"]

# Read TCP metrics such as established, time wait and sockets counts.
[[inputs.netstat]]
  # no configuration

# Read metrics about system load & uptime
[[inputs.system]]
  ## Uncomment to remove deprecated metrics.
  # fielddrop = ["uptime_format"]

# Read metrics from one or many mysql servers
[[inputs.mysql]]
  servers = ["tcp(127.0.0.1:3306)/"]

$> kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
ftps-75f885d4c4-4knkr         1/1     Running   0          36m
grafana-644575c45-qrx24       1/1     Running   0          36m
influxdb-7cdd559dc7-r5lf4     1/1     Running   0          33m
mysql-5c4c9b7b9b-wj66g        1/1     Running   0          36m
nginx-7f4577bf74-hbvt5        1/1     Running   0          36m
phpmyadmin-76d78558db-crssd   1/1     Running   0          36m
wordpress-75b6bd5576-pflrr    1/1     Running   0          36m
$> kubectl exec -it influxdb-7cdd559dc7-r5lf4 -- sh
/ # influx
Connected to http://localhost:8086 version 1.8.3
InfluxDB shell version: 1.8.3
> show databases
name: databases
name
----
influxdb
wordpress
phpmyadmin
nginx
ftps
grafana
mysql
_internal

Grafana

$> kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
ftps-75f885d4c4-4knkr         1/1     Running   0          91m
grafana-644575c45-bz2n6       1/1     Running   0          11m
influxdb-7cdd559dc7-r5lf4     1/1     Running   0          88m
mysql-5c4c9b7b9b-wj66g        1/1     Running   0          91m
nginx-7f4577bf74-hbvt5        1/1     Running   0          91m
phpmyadmin-76d78558db-crssd   1/1     Running   0          91m
wordpress-75b6bd5576-pflrr    1/1     Running   0          91m
$> kubectl cp grafana-644575c45-bz2n6:/usr/share/grafana/data/grafana.db ./grafana.db

$> kubectl cp ./grafana.db grafana-644575c45-p9tln:/usr/share/grafana/data/grafana.db

쿠버네티스 대시보드 접속

$> kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

$> kubectl create serviceaccount dashboard-admin-sa
serviceaccount/dashboard-admin-sa created
$> kubectl create clusterrolebinding dashboard-admin-sa
--clusterrole=cluster-admin --serviceaccount=default:dashboard-admin-sa
Error: required flag(s) "clusterrole" not set
zsh: command not found: --clusterrole=cluster-admin
$> kubectl create clusterrolebinding dashboard-admin-sa --clusterrole=cluster-admin --serviceaccount=default:dashboard-admin-sa
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin-sa created
$> kubectl get secrets
NAME                             TYPE                                  DATA   AGE
dashboard-admin-sa-token-lqn75   kubernetes.io/service-account-token   3      26s
default-token-djwdx              kubernetes.io/service-account-token   3      7m41s
$> kubectl describe secret dashboard-admin-sa-token-lqn75
Name:         dashboard-admin-sa-token-lqn75
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin-sa
              kubernetes.io/service-account.uid: dd42c0b3-ae0a-4d1c-a48c-1963077b4f80

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1111 bytes
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Im1Sdnp3d3RMNmtNZ0RCTnJtc3J5WWJ3eEp5Y0cwdG5aLXB0VnQxTkNLSE0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRhc2hib2FyZC1hZG1pbi1zYS10b2tlbi1scW43NSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkZDQyYzBiMy1hZTBhLTRkMWMtYTQ4Yy0xOTYzMDc3YjRmODAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpkYXNoYm9hcmQtYWRtaW4tc2EifQ.1UOnlI0xo3v_w0HMFfJFaqxEVV0wZRzsVSPyk-QfVxn4qOWN09O1WrAQLE0OhuVs2F1U1RtEV6wyi1Jwo1DVe8uRl4oKXYSJlPx64nvSUCsBfRHCzRC0Cjr2672dRDnouF1GLWkAQQ86TiRQ_Q3kTdxzAy9Sh9CCds3D-chx4VxWmrK0pCk9QQ_zECg3pc4lZIGgxxcT0hDhvr0GKxcqMmOmj8Swrzu4MpCBBVNWZY5NkxhA7NJfr9NqZKT3E_nWK4a0cQva1Vi31t0ErjfD7I3Y13eCNBGjoi97qSXfcFkc59goraZV_s5drFpmrPjDxK5WVgxZkLfbQHivUpDThQ

디플로이먼트 퍼시스턴스 강건화

• livenessProbe : 컨테이너 내부 프로세스가 동작 중인지 주기적으로 확인한다. 요청에 대한 응답이 없다면 컨테이너를 재시작한다.

• readinessProbe : 컨테이너 내부 프로세스가 요청을 받을 수 있는지를 확인한다. 요청을 받을 수 없다면 해당 컨테이너의 IP를 제거해 요청을 받을 수 없도록 한다.

• startupProbe : 컨테이너 내부 프로세스가 시작되었는지 확인한다. 시작되지 않았다면 재시작 한다.

어떤 컨테이너에 대해 적용하나?