Log【2022/12】

Otthertune[2022/12]

Ref

• blog ottertune 配置记录
• rabbitmq tutorial
• Python 3.6.13

Sum

Linux Usage

• check thread ps aux --sort=lstart

Mysql

• mysql install

  sudo apt-get install mysql-server
  apt install mysqlclient
  apt-get install libmysqlclient-dev
  pip install mysqlclient==1.3.12
  sudo apt-get install python-pip python-dev python-mysqldb rabbitmq-server
  # usr:root passwd:xiaotong 
  

• mysql use:

  mysql -u root -p
  # passwd xiaotong
  

Postgresql

• download from tar.gz website and decompress

  tar -xvf [.tar.gz]
  cd []
  

• compile and build

  apt-get install libreadline5 # readline
  apt-get install libreadline6-dev # readline
  apt-get install zlib1g.dev # zlib
  ./configure --prefix=/usr/pgsql9.6/ #prefix means the install path
  make
  make install
  

  then pgsql9.6 is installed at /usr/pgsql9.6
• init postgres see blog

  cd /usr/pgsql9.6
  mkdir data
  adduser postgres # passwd: postgres
  chown postgres data
  su postgres
  cd /usr/pgsql9.6
  bin/initdb -D data
  # bin/pg_ctl -D data -l logfile start
  bin/pg_ctl -D data start
  bin/createdb test
  bin/psql test
  

• start postgres see blog

  # bin/pg_ctl -D data -l logfile start
  bin/pg_ctl -D data start
  bin/createdb test
  bin/psql test
  

• postgres use:

  su postgres # passwd postgres
  /usr/pgsql9.6/bin/psql
  

Docker

docker build -t ottertune-base -f ./docker/Dockerfile.base .
docker build -t ottertune-driver -f ./docker/Dockerfile.driver .
docker build -t ottertune-web -f ./docker/Dockerfile.web .

docker imgages # see image
docker rmi -f [image_id]
docker ps -a # see container
docker rm -f [container_id]

docker-compose -f docker-compose.build.yml build # build image
docker-compose -f docker-compose.yml up # build container
docker inspect --format '{{.LogPath}}' [container id] #return log file can see log file via `vim`

cd /root/ottertune/
bash install_images.sh
cd docker
BACKEND=postgresql WEB_ENTRYPOINT="''" sh create-docker-compose.sh $dcfile_postgres # from .travis.yml
docker-compose -f docker-compose.yml up

docker cp [source file name] [container id]:[target file name]

docker exec -it [container name] /bin/bash

Celery

• asynchronous message queueing
• celery = message broker + worker + task result store
• work with rabbitmq/reddis
  • with reddis see celery blog

    tar+make install redis-6.2.7 
    pip install redis==2.10.6
    pip install celery==3.1.24
    # celery_task.py --> `celery worker -A celery_task -l info`
    # produce_task.py --> `python produce_task.py`
    # result.py --> python result.py --id [id get from produce_task.py]
    

    tutorial video
  • with rabbitmq see rabbitmq tutorial

    python receive.py
    python send.py
    

RabbitMQ

• stop app rabbitmqctl stop_app
• see queue list status rabbitmqctl list_queues
• see status rabbitmqctl status
• clear all queues in rabbitmq rabbitmqctl reset
• clear item in rabbitmq rabbitmqctl purge_queue celery
• restart rabbitmq bash rabbitmq_restart.sh

  rabbitmqctl stop_app
  rabbitmqctl reset
  rabbitmqctl start_app
  rabbitmqctl list_queues
  

Ottertune

• debug ottertune

  # in /root/ottertune/server/website
  fab -f script/management/fabfile.py start_debug_server
  # in /root/ottertune/client/driver
  fab -f /root/ottertune/client/driver/fabfile.py loop:i=10
  # check port usage
  netstat -anp | grep 8000
  # put breakpoint
  rdb.set_trace() # tong 11/29
  # telnet 
  telnet localhost [port] # port may change
  

Django

• version dependency

Philosophy

• FIND A PROPER START POINT
• version dependency 99%
• ACID: Atomic (rerun the whole pipeline if err)

Ottertune[2022/11]

🌟see blog

env set log

python=3.6.13

project 🌟

Goal: AI tune DB

archi

• ottertune = server + client
• server = mysql (store training data) + Django (UI) + Celery (ML task schedule)
• client = target dbms + controller + driver
  controller control target dmbs
  driver control controller
• [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-YiPaygWa-1670479625855)(image/2.png)]

env set

• ottertune = server set + client set
  [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-PZLuBSgt-1670479625856)(image/638051-20190529035020576-798495823.png)]
• need oltpbench for create wkld and run wkld get result

oltpbench

• generate and run wkld --> get performance
• otlpbenchmark git repo
• ./oltpbenchmark -b tpcc -c config/tpcc_config.xml --create=true --load=true -s 10 -o outputfile
• result of cmd
  [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-OhdQ9qah-1670479625856)(image/3.png)]

Ref

1. OtterTune源码解析
2. otlpbenchmark git repo
3. ottertune git repo

Sum 🌟

python install

• see available python and do something with them
  Run python and see the version of it. You can also check the folder /usr/local/bin or /usr/bin since python is always installed there.
  python import sys print(sys.executable)

• get pakage from website see 菜鸟教程python安装

• choose a proper version (ps: the hardest part)
  We take python3.7.0 as example

• compile and build (ps: python depends on gcc)

  ./configure
  make
  make install
  

• build soft link (ps: for mvc)

  ln -s /usr/bin/python /usr/local/bin/python3.7 # when type `python`, you actually run `python3.7` useful for mvc
  ln -s /usr/local/bin/python3.7 /usr/bin/python
  

• try and test: make sure you really install the python properly

• how to choose python version❓

  It depends on the pakcage that your project need --> get info from repo or try (ps: I am confused with this part😭)

network 🌟

• do this part first❗️
• basic idea: network need dns & domestic image
• set dns set dns
• Warning: You can only have 3 dns ip❗️
  

  vim /etc/resolv.conf # tmp
  vim /etc/resolvconf/resolv.conf.d/base # perm   
  # nameserver 10.3.3.3
  # nameserver 114.114.114.114
  # nameserver 8.8.8.8 
  

• domestic image

  vim /etc/apt/sources.list
  # add the proper content from [网络设置清华源](https://mirror.tuna.tsinghua.edu.cn/help/ubuntu/)
  

• reboot the server

pip install

• sudo apt-get install pip
• python get-pip.py (ps: recommend)
  Install get-pip.py from website and run python get-pip.py.
• install pip from pypi
  website contains python’s all pakcages, which inlude pip. You can install pip like other pakages.

pakage install

• pip install [pakage name] （ps: need pip)
• pip install [pakage name] -i [domestic image]

  you can choose domestic image from below
  https://pypi.tuna.tsinghua.edu.cn/simple/	# 清华大学
  https://mirrors.aliyun.com/pypi/simple/		# 阿里云
  https://pypi.douban.com/simple/				# 豆瓣
  https://pypi.mirrors.ustc.edu.cn/simple/	# 中国科学技术大学
  https://pypi.hustunique.com/	
  # ————————————————
  # 版权声明：本文为优快云博主「Doris404」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
  # 原文链接：https://blog.youkuaiyun.com/kullollo/article/details/109448254
  

• install pakage.tar.gz from pypi and run cd [pakage] & python setup.py install
• install pakage from pakage.whl in pypi and run pip install [pakage.whl]

Philosophy

goal first

• don’t update software frequently (ps: update pip)
• see only the important info (ps: warning is ok, useless err is ok)
  [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-E9YUU5Sw-1670479625856)(image/4.png)]
  easy first
• passwd free login how to set passwd free login

  ssh-copy-id -i /Users/xiaotong/.ssh/id_rsa.pub root@[ip]
  

log and checkpoint

• this README.md
• env set will last long time log and checkpoint will keep you clear