We’re integrating Tectonic with Red Hat OpenShift

We are bringing the best of Tectonic to Red Hat OpenShift to build the most secure, hybrid Kubernetes application platform.

Troubleshooting etcd nodes using SSH

Tectonic etcd nodes are not assigned a public IP address, only the master node are. To debug an etcd node, SSH to it through a master (bastion host) or use a VPN connected to the internal network.

To do so, perform the following:

Set up SSH agent forwarding

Once a passphrase of the local ssh key is added to ssh-agent, you will not be prompted for the credentials the next time connecting to nodes via SSH or SCP. The following instructions outline adding a passphrase to the ssh-agent on the system.

  1. At the terminal, enter:

$ eval ssh-agent

  1. Run the following:

$ ssh-add

The ssh-add command prompts for a private key passphrase and adds it to the list maintained by ssh-agent.

  1. Enter your private key passphrase.

  2. Before logging out, run the following:

$ kill $SSH_AGENT_PID

To automatically run this command when logging out, place it in the .logout file if you are using csh or tcsh. Place the command in the .bash_logout file if you are using bash.

Connect to a master node

SSH to a master node with its EXTERNAL-IP, providing the -A flag to forward the local ssh-agent. Add the -i option giving the location of the ssh key known to Tectonic:

$ ssh -A core@10.0.23.37 -i /path/to/tectonic/cluster/ssh/key

Get the IP address of the etcd nodes

Run the following command on the master instance:

core@ip-10-0-23-37 ~ $ grep etcd /opt/tectonic/manifests/kube-apiserver.yaml 
        - --etcd-servers=http://10.0.23.31:2379

Connect to an etcd node

# From the master node
$ ssh core@10.0.23.31

To investigate issues with etcd, execute:

$ systemctl status etcd-member && journalctl etcd-member