doc/source/user/adding-and-removing-hosts.rst - openstack/kolla-ansible - Gitiles

 =========================
 Adding and removing hosts
 =========================

 This page discusses how to add and remove nodes from an existing cluster. The
 procedure differs depending on the type of nodes being added or removed, which
 services are running, and how they are configured. Here we will consider two
 types of nodes - controllers and compute nodes. Other types of nodes will need
 consideration.

 Any procedure being used should be tested before being applied in a production
 environment.

 Adding new hosts
 ================

 .. _adding-new-controllers:

 Adding new controllers
 ----------------------

 The :doc:`bootstrap-servers command
 </reference/deployment-and-bootstrapping/bootstrap-servers>` can be used to
 prepare the new hosts that are being added to the system.  It adds an entry to
 ``/etc/hosts`` for the new hosts, and some services, such as RabbitMQ, require
 entries to exist for all controllers on every controller. If using a
 ``--limit`` argument, ensure that all controllers are included, e.g. via
 ``--limit control``. Be aware of the :ref:`potential issues <rebootstrapping>`
 with running ``bootstrap-servers`` on an existing system.

 .. code-block:: console

    kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]

 Pull down container images to the new hosts. The ``--limit`` argument may be
 used and only needs to include the new hosts.

 .. code-block:: console

    kolla-ansible -i <inventory> pull [ --limit <limit> ]

 Deploy containers to the new hosts. If using a ``--limit`` argument, ensure
 that all controllers are included, e.g. via ``--limit control``.

 .. code-block:: console

    kolla-ansible -i <inventory> deploy [ --limit <limit> ]

 The new controllers are now deployed. It is recommended to perform testing
 of the control plane at this point to verify that the new controllers are
 functioning correctly.

 Some resources may not be automatically balanced onto the new controllers. It
 may be helpful to manually rebalance these resources onto the new controllers.
 Examples include networks hosted by Neutron DHCP agent, and routers hosted by
 Neutron L3 agent. The `removing-existing-controllers`_ section provides an
 example of how to do this.

 .. _adding-new-compute-nodes:

 Adding new compute nodes
 ------------------------

 The :doc:`bootstrap-servers command
 </reference/deployment-and-bootstrapping/bootstrap-servers>`, can be used to
 prepare the new hosts that are being added to the system.  Be aware of the
 :ref:`potential issues <rebootstrapping>` with running ``bootstrap-servers`` on
 an existing system.

 .. code-block:: console

    kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]

 Pull down container images to the new hosts. The ``--limit`` argument may be
 used and only needs to include the new hosts.

 .. code-block:: console

    kolla-ansible -i <inventory> pull [ --limit <limit> ]

 Deploy containers on the new hosts. The ``--limit`` argument may be used and
 only needs to include the new hosts.

 .. code-block:: console

    kolla-ansible -i <inventory> deploy [ --limit <limit> ]

 The new compute nodes are now deployed. It is recommended to perform
 testing of the compute nodes at this point to verify that they are functioning
 correctly.

 Server instances are not automatically balanced onto the new compute nodes. It
 may be helpful to live migrate some server instances onto the new hosts.

 .. code-block:: console

    openstack server migrate <server> --live-migration --host <target host> --os-compute-api-version 2.30

 Alternatively, a service such as :watcher-doc:`Watcher </>` may be used to do
 this automatically.

 Removing existing hosts
 =======================

 .. _removing-existing-controllers:

 Removing existing controllers
 -----------------------------

 When removing controllers or other hosts running clustered services, consider
 whether enough hosts remain in the cluster to form a quorum. For example, in a
 system with 3 controllers, only one should be removed at a time. Consider also
 the effect this will have on redundancy.

 Before removing existing controllers from a cluster, it is recommended to move
 resources they are hosting. Here we will cover networks hosted by Neutron DHCP
 agent and routers hosted by Neutron L3 agent. Other actions may be necessary,
 depending on your environment and configuration.

 For each host being removed, find Neutron routers on that host and move them.
 Disable the L3 agent. For example:

 .. code-block:: console

    l3_id=$(openstack network agent list --host <host> --agent-type l3 -f value -c ID)
    target_l3_id=$(openstack network agent list --host <target host> --agent-type l3 -f value -c ID)
    openstack router list --agent $l3_id -f value -c ID | while read router; do
      openstack network agent remove router $l3_id $router --l3
      openstack network agent add router $target_l3_id $router --l3
    done
    openstack network agent set $l3_id --disable

 Repeat for DHCP agents:

 .. code-block:: console

    dhcp_id=$(openstack network agent list --host <host> --agent-type dhcp -f value -c ID)
    target_dhcp_id=$(openstack network agent list --host <target host> --agent-type dhcp -f value -c ID)
    openstack network list --agent $dhcp_id -f value -c ID | while read network; do
      openstack network agent remove network $dhcp_id $network --dhcp
      openstack network agent add network $target_dhcp_id $network --dhcp
    done

 Stop all services running on the hosts being removed:

 .. code-block:: console

    kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]

 Remove the hosts from the Ansible inventory.

 Reconfigure the remaining controllers to update the membership of clusters such
 as MariaDB and RabbitMQ. Use a suitable limit, such as ``--limit control``.

 .. code-block:: console

    kolla-ansible -i <inventory> deploy [ --limit <limit> ]

 Perform testing to verify that the remaining cluster hosts are operating
 correctly.

 For each host, clean up its services:

 .. code-block:: console

    openstack network agent list --host <host> -f value -c ID | while read id; do
      openstack network agent delete $id
    done

    openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID | while read id; do
      openstack compute service delete --os-compute-api-version 2.53 $id
    done

 .. _removing-existing-compute-nodes:

 Removing existing compute nodes
 -------------------------------

 When removing compute nodes from a system, consider whether there is capacity
 to host the running workload on the remaining compute nodes. Include overhead
 for failures that may occur.

 Before removing compute nodes from a system, it is recommended to migrate or
 destroy any instances that they are hosting.

 For each host, disable the compute service to ensure that no new instances are
 scheduled to it.

 .. code-block:: console

    openstack compute service set <host> nova-compute --disable

 If possible, live migrate instances to another host.

 .. code-block:: console

    openstack server list --host <host> -f value -c ID | while read server; do
      openstack server migrate --live-migration $server
    done

 Verify that the migrations were successful.

 Stop all services running on the hosts being removed:

 .. code-block:: console

    kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]

 Remove the hosts from the Ansible inventory.

 Perform testing to verify that the remaining cluster hosts are operating
 correctly.

 For each host, clean up its services:

 .. code-block:: console

    openstack network agent list --host <host> -f value -c ID | while read id; do
      openstack network agent delete $id
    done

    openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID | while read id; do
      openstack compute service delete --os-compute-api-version 2.53 $id
    done
	=========================
	Adding and removing hosts
	=========================

	This page discusses how to add and remove nodes from an existing cluster. The
	procedure differs depending on the type of nodes being added or removed, which
	services are running, and how they are configured. Here we will consider two
	types of nodes - controllers and compute nodes. Other types of nodes will need
	consideration.

	Any procedure being used should be tested before being applied in a production
	environment.

	Adding new hosts
	================

	.. _adding-new-controllers:

	Adding new controllers
	----------------------

	The :doc:`bootstrap-servers command
	</reference/deployment-and-bootstrapping/bootstrap-servers>` can be used to
	prepare the new hosts that are being added to the system. It adds an entry to
	``/etc/hosts`` for the new hosts, and some services, such as RabbitMQ, require
	entries to exist for all controllers on every controller. If using a
	``--limit`` argument, ensure that all controllers are included, e.g. via
	``--limit control``. Be aware of the :ref:`potential issues <rebootstrapping>`
	with running ``bootstrap-servers`` on an existing system.

	.. code-block:: console

	kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]

	Pull down container images to the new hosts. The ``--limit`` argument may be
	used and only needs to include the new hosts.

	.. code-block:: console

	kolla-ansible -i <inventory> pull [ --limit <limit> ]

	Deploy containers to the new hosts. If using a ``--limit`` argument, ensure
	that all controllers are included, e.g. via ``--limit control``.

	.. code-block:: console

	kolla-ansible -i <inventory> deploy [ --limit <limit> ]

	The new controllers are now deployed. It is recommended to perform testing
	of the control plane at this point to verify that the new controllers are
	functioning correctly.

	Some resources may not be automatically balanced onto the new controllers. It
	may be helpful to manually rebalance these resources onto the new controllers.
	Examples include networks hosted by Neutron DHCP agent, and routers hosted by
	Neutron L3 agent. The `removing-existing-controllers`_ section provides an
	example of how to do this.

	.. _adding-new-compute-nodes:

	Adding new compute nodes
	------------------------

	The :doc:`bootstrap-servers command
	</reference/deployment-and-bootstrapping/bootstrap-servers>`, can be used to
	prepare the new hosts that are being added to the system. Be aware of the
	:ref:`potential issues <rebootstrapping>` with running ``bootstrap-servers`` on
	an existing system.

	.. code-block:: console

	kolla-ansible -i <inventory> bootstrap-servers [ --limit <limit> ]

	Pull down container images to the new hosts. The ``--limit`` argument may be
	used and only needs to include the new hosts.

	.. code-block:: console

	kolla-ansible -i <inventory> pull [ --limit <limit> ]

	Deploy containers on the new hosts. The ``--limit`` argument may be used and
	only needs to include the new hosts.

	.. code-block:: console

	kolla-ansible -i <inventory> deploy [ --limit <limit> ]

	The new compute nodes are now deployed. It is recommended to perform
	testing of the compute nodes at this point to verify that they are functioning
	correctly.

	Server instances are not automatically balanced onto the new compute nodes. It
	may be helpful to live migrate some server instances onto the new hosts.

	.. code-block:: console

	openstack server migrate <server> --live-migration --host <target host> --os-compute-api-version 2.30

	Alternatively, a service such as :watcher-doc:`Watcher </>` may be used to do
	this automatically.

	Removing existing hosts
	=======================

	.. _removing-existing-controllers:

	Removing existing controllers
	-----------------------------

	When removing controllers or other hosts running clustered services, consider
	whether enough hosts remain in the cluster to form a quorum. For example, in a
	system with 3 controllers, only one should be removed at a time. Consider also
	the effect this will have on redundancy.

	Before removing existing controllers from a cluster, it is recommended to move
	resources they are hosting. Here we will cover networks hosted by Neutron DHCP
	agent and routers hosted by Neutron L3 agent. Other actions may be necessary,
	depending on your environment and configuration.

	For each host being removed, find Neutron routers on that host and move them.
	Disable the L3 agent. For example:

	.. code-block:: console

	l3_id=$(openstack network agent list --host <host> --agent-type l3 -f value -c ID)
	target_l3_id=$(openstack network agent list --host <target host> --agent-type l3 -f value -c ID)
	openstack router list --agent $l3_id -f value -c ID \| while read router; do
	openstack network agent remove router $l3_id $router --l3
	openstack network agent add router $target_l3_id $router --l3
	done
	openstack network agent set $l3_id --disable

	Repeat for DHCP agents:

	.. code-block:: console

	dhcp_id=$(openstack network agent list --host <host> --agent-type dhcp -f value -c ID)
	target_dhcp_id=$(openstack network agent list --host <target host> --agent-type dhcp -f value -c ID)
	openstack network list --agent $dhcp_id -f value -c ID \| while read network; do
	openstack network agent remove network $dhcp_id $network --dhcp
	openstack network agent add network $target_dhcp_id $network --dhcp
	done

	Stop all services running on the hosts being removed:

	.. code-block:: console

	kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]

	Remove the hosts from the Ansible inventory.

	Reconfigure the remaining controllers to update the membership of clusters such
	as MariaDB and RabbitMQ. Use a suitable limit, such as ``--limit control``.

	.. code-block:: console

	kolla-ansible -i <inventory> deploy [ --limit <limit> ]

	Perform testing to verify that the remaining cluster hosts are operating
	correctly.

	For each host, clean up its services:

	.. code-block:: console

	openstack network agent list --host <host> -f value -c ID \| while read id; do
	openstack network agent delete $id
	done

	openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID \| while read id; do
	openstack compute service delete --os-compute-api-version 2.53 $id
	done

	.. _removing-existing-compute-nodes:

	Removing existing compute nodes
	-------------------------------

	When removing compute nodes from a system, consider whether there is capacity
	to host the running workload on the remaining compute nodes. Include overhead
	for failures that may occur.

	Before removing compute nodes from a system, it is recommended to migrate or
	destroy any instances that they are hosting.

	For each host, disable the compute service to ensure that no new instances are
	scheduled to it.

	.. code-block:: console

	openstack compute service set <host> nova-compute --disable

	If possible, live migrate instances to another host.

	.. code-block:: console

	openstack server list --host <host> -f value -c ID \| while read server; do
	openstack server migrate --live-migration $server
	done

	Verify that the migrations were successful.

	Stop all services running on the hosts being removed:

	.. code-block:: console

	kolla-ansible -i <inventory> stop --yes-i-really-really-mean-it [ --limit <limit> ]

	Remove the hosts from the Ansible inventory.

	Perform testing to verify that the remaining cluster hosts are operating
	correctly.

	For each host, clean up its services:

	.. code-block:: console

	openstack network agent list --host <host> -f value -c ID \| while read id; do
	openstack network agent delete $id
	done

	openstack compute service list --os-compute-api-version 2.53 --host <host> -f value -c ID \| while read id; do
	openstack compute service delete --os-compute-api-version 2.53 $id
	done