Software » EEA website systems

= How to update the EEA website on HA cluster =
There are mainly two ways to perform an update on the HA cluster. The most common one is the "Standard update", this is done when standard plone/zope products are updated. The "Major update" procedure is done only when main components/infrastructure need to be updated on one or both servers, e.g. of such are ZEO, plone version, squid, apache, OS updates and hardware updates.

==== Servers ==== * '''Frontend:''' * parrot * cockatoo * '''Backend:''' * rocksquirrel * redsquirrel * graysquirrel

==== '''Development portal on Unicorn''' ==== * URL: '''http://www-devel.webdev.eea.europa.eu''' * Path on file system: '''/var/eeawebtest/www.eea.europa.eu.DEVEL''' * CFG file: '''unicorn-devel.cfg''' * Only '''www1''' is mapped under Apache

==== '''Pre-production portal on Unicorn''' ==== * URL: '''http://www-pre-production.webdev.eea.europa.eu''' * Path on file system: '''/var/eeawebtest/www.eea.europa.eu.PRE-PRODUCTION''' * CFG file: '''unicorn-pre-production.cfg''' * Restart Zeo server * After svn up and rebuild to restart all instances the following script can be used: '''/var/eeawebtest/www.eea.europa.eu.PRE-PRODUCTION/bin/unicorn-pre-production-restart''' * This portal has a running Varnish in front (also found on Unicorn)

Pre-production update ==== '''Prerequisite''' ====
''' Before any update on productions you must have done the full update, deployment and tests on pre-production environment (hudson and/or unicorn).''' 1. Check on '''Jenkins''' that all builds are green ([http://ci.eionet.europa.eu/ http://ci.eionet.europa.eu])
1. On any squirrel run '''/tools/releases-candidates.sh''' '''> release.candidates.txt''' in order to detect packages to be released
1. '''Release''' all '''eggs''' detected above (see more under [wiki:HowToReleaseAnEgg here])
1. On unicorn under '''/var/eeawebtest/www.eea.europa.eu.PRE-PRODUCTION''':
1. make sure that there are no local modification (svn st)
1. svn up
1. run buildout
1. restart ZEO
1. re-start one instance in fg to check zope starts
1. stop the instance you have in fg
1. stop all instance using ./bin/unicorn-pre-production-stop
1. restart all instaces using ./bin/unicorn-pre-production-restart
1. [optional] invalidate varnish cache: service varnish restart
1. under ZMI regenerate all resources css/js/kss
1. check under''' http://www-pre-production.webdev.eea.europa.eu '''that portal looks fine Standard Production update
Used for normal upgrade of website products, e.g. we have new eggs to deploy. Backend servers are '''rocksquirrel''', '''redsquirrel''' and '''graysquirrel'''.

==== Update backends: ====
1. '''INFORM''' on developers in chat room about "HA Production upgrade", get acknowledgement from server admin.
1. Add '''portal message''' to inform CMS users an upgrade is on the way (see template under /custom_admin)
1. '''SSH''' to one server, and become root, cd /var/eea-buildout-plone4 directory
1. make sure that there are no local modification (svn st), if so further clarification of changes is requested in order to proceed.
1. run '''/tools/releases-candidates.sh''' in order to create the release-notes-YYYY-MM-DD.txt to be committed at the end of this procedure. In this file you will also need to scan for any "Manual update step" needed for some packages.
1. '''svn up'''
1. Run '''buildout''' (buildout -c backend.cfg)
1. Pre-check with '''serviceWWW''' fg everything starts fine
1. Do the '''steps 4 to 6''' on all remaining backend servers
1. '''Stop and start''' all instances and wait a little between each command (use '''crm_mon'''):
1. for i in {1,4}; do /usr/sbin/crm resource stop www$i'P4'-clones ;done
1. for i in {1,4}; do /usr/sbin/crm resource start www$i'P4'-clones ;done
1. for i in {3,6}; do /usr/sbin/crm resource stop www$i'P4'-clones ;done
1. for i in {3,6}; do /usr/sbin/crm resource start www$i'P4'-clones ;done
1. for i in {2,5}; do /usr/sbin/crm resource stop www$i'P4'-clones ;done
1. for i in {2,5}; do /usr/sbin/crm resource start www$i'P4'-clones ;done
1. '''Restart''' '''serviceWWW''' on each server
1. '''Check all instances have restarted''' by listing the zope socketfiles and verifying that creation time is current: ls -la /var/eea-buildout-plone4/var/zopectlsock-www*
1. '''Check each instance''' on all backend servers (including serviceWWW) via direct port number has restarted properly. Use the convenient bash script in tools/responsecheck.sh will output response code for each instance on all squirrels. restart any instance which does not return 200.
1. '''ZMI settings and upgrades'''. Login to one instance via zmi, follow any manual steps found in the '''changes file '''(search for "Upgrade step:")
1. '''Cleanup custom'''. Remove anything in custom that has been included in this update
1. '''Invalidate''' templates, css, js and kss. This is done by resaving portal_css, portal_javascripts and portal_kss
1. Check the EEA '''frontpage''' loads correctly (both as anonymous as authenticated)
1. Ask '''each developer''' to double check their latest update are working correctly
1. '''Check external sites''' like glossary and eunis are using latest template css/js
1. '''Commit the changes file created at the beginning''' report file under /eea-buildout/docs/release-notes-YYYY-MM-DD.txt. * Commit with message having the refs to all the tickets included in the release notes, use the following command to get a list of unique tickets numbers: * {{{
grep -o '#[0-9][0-9][0-9][0-9]' release-notes-2012-02-15.txt | awk -F, '!_[$1]++' | sed -e :a -e N -e 's/\n/,/' -e ta
}}}
1. Disable the '''portal message''' via ZMI custom_admin
1. Release EEA packages on '''pypi''' and '''plone.org''', see more info [wiki:HowToReleaseAnEgg#HowdoIreleaseeggsonpypiorplone.org here]

If any manual upgrade steps was not done please report it in this file as TODO.

==== Update frontends (Apache rewrite rules and/or Varnish): ====
If you need to update the frontend Apache and Varnish you need to do the svn up, '''chown of files''' and apache restart/reload on the two frontend servers ('''parrot''', '''cockatoo''') : {{{
kill -HUP `ps axu |grep httpd|grep root|grep -v grep| gawk '{print $2}'`
}}} * For Apache restart please use crm commands to stop and start the resource {{{
crm resource restart ClusterApache
}}}

• For Apache configration reload please execute on each server the following:

Command reference ==== The command to manage/unmanage a resource ==== {{{
crm resource unmanage <<resource>>
crm resource manage <<resource>>
}}}