Here we could not mentioned grpc etcd load balance, what we want to focus if that how to smooth upgrade/ offline one host/one service by etcd.

In order to reach this goal, below is the cases to be verified by us.

1. verify two myservice service register etcd, and could round robin balance for requests

     start services:

         myservice service 1: port 9000

         myservice servie 2 : port 9001

     test:

     test programmer to send request to myservice service by etcd, two request, one is processed by 9000, the other one is processed by 9001 

     pass

2. manually delete one one node from etcd:

    myservice service still be alive, but no serve new request

      -- pass

    myservice service not register into etcd again if not restart this service

      -- pass

    myservice service could service request which has been in queue

      test steps: add time.sleep, then send request, erase service node from etcd

      -- pass

3. view myservice etcd register key 

      ./etcdctl --endpoints=$host1:2379,$host2:2379,$host3:2379 get --prefix myservice

4. verify erase scipt to delete myservice service etcd register key 

action steps:

sh remove_service_from_etcd.sh -p ./ -e $host1:2379,$host2:2379,$host3:2379 -f myservice-service -s myservice13.net:9000  -t 1

5. Stop myservice service auto start on machine restart

in case that, ops do debug on machine after restart, when ops say it is fine, start relevant service to make it service request

6. remove_service_from_etcd.sh

Here provide a script remove_service_from_etcd.sh for your reference.

#!/bin/bash

function usage() {

    echo "  sh remove_service_from_etcd.sh: options"

    echo "  -p ectd install path"

    echo "  -e etcd ENDPOINTS, for example: http://HOST_01:2379,http://HOST_02:2379,http://HOST_03:2379"

    echo "  -f service etcd prefix, for example: myservice-service"

    echo "  -s service tag, for example: myservice01.net:9000"

    echo "  -t time out time for myservice service request, for example: 100s"

    echo "  -d service name: for example: test"

    echo "  -h help info"

    echo "  Eg:"

    echo "    sh remove_service_from_etcd.sh -p /opt/cli/etcd-cluster/current -e http://$HOST_07:2379,http://$HOST_08:2379 -f myservice-service -s myservice01.net:9000 -d test -t 1s"

    exit

}

if [ $# -eq 0 ]; then

  usage

fi

ectd_install_path=""

ENDPOINTS=""

prefix=""

myservice_service_tag=""

my_service_name=""

timeout=""

while getopts p:e:f:s:t:d:h opt

do

 case $opt in

 p) echo "-p etcd install path $OPTARG"

     ectd_install_path=$OPTARG

     ;;

 e) echo "-e etcd entrypoints $OPTARG"

     ENDPOINTS=$OPTARG

     ;;

 f) echo "-f service etcd prefix $OPTARG"

     prefix=$OPTARG

     ;;

 s) echo "-s service tag $OPTARG"

     myservice_service_tag=$OPTARG

     ;;

 t) echo "-t time out time $OPTARG"

     timeout=$OPTARG

     ;;

 d) echo "-d service name $OPTARG"

     my_service_name=$OPTARG

     ;;

 h) echo "usage instruction"

     usage

     ;;

 *) echo $opt not a option

     usage

     ;;

 esac

done

if [ "$ectd_install_path" == "" ]

then

    echo "ectd_install_path must specify"

    exit

fi

if [ "$ENDPOINTS" == "" ]

then

    echo "ENDPOINTS must specify"

    exit

fi

if [ "$prefix" == "" ]

then

    echo "prefix must specify, for example:myservice-service"

    exit

fi

if [ "$myservice_service_tag" == "" ]

then

    echo "myservice_service_tag must specify, for example:myservice09.net:9000"

    exit

fi

if [ "$timeout" == "" ]

then

    echo "timeout must specify"

    exit

fi

export PATH=$ectd_install_path:$PATH

export ETCDCTL_API=3

echo "Check the service availability"

response=`etcdctl --endpoints=$ENDPOINTS put foo bar`

if [ $response != "OK" ]

then

    echo "ETCDCTL_API=3 etcdctl --endpoints=$ENDPOINTS put foo bar check fail!!!"

    exit

fi

if [ "$my_service_name" == "" ]

then

    etcdctl--endpoints=$ENDPOINTS get --prefix $prefix | grep $prefix | grep $myservice_service_tag |  xargs-l -p etcdctl --endpoints=$ENDPOINTS del $i

else

    etcdctl--endpoints=$ENDPOINTS get --prefix $prefix | grep $prefix | grep $myservice_service_tag | grep $my_service_name | xargs -l -p etcdctl --endpoints=$ENDPOINTS del $i

fi

sleep $timeout

echo "+++++++++++++++++++++++++++++++++"

echo " Eliminated service will not service any request"

echo "+++++++++++++++++++++++++++++++++"

7. start etcd service

#!/usr/bin/env bash

function start_service() {

    /usr/local/bin/etcd--name testetcd --initial-advertise-peer-urls http://0.0.0.0:2380 --listen-peer-urls http://0.0.0.0:2380 --advertise-client-urls http://0.0.0.0:2379 -listen-client-urls http://0.0.0.0:2379 --initial-cluster testetcd=http://0.0.0.0:2380 --initial-cluster-state new -initial-cluster-token token-1 &

}

start_service

exec $@

Reference

https://futurestud.io/tutorials/coreos-read-write-values-to-etcd

https://github.com/etcd-io/etcd