Posted by cookiesunshinex on September 29, 2010 at 9:53am
After running $ sudo /usr/local/bin/update_mercury.sh for the first time, Apache Solr is unable to connect to server.
After further investigation, I find that Tomcat6 is not running.
ubuntu@dev:~$ sudo /etc/init.d/tomcat6 status
* Tomcat servlet engine is not running.
ubuntu@dev:~$ sudo /etc/init.d/tomcat6 start
* Starting Tomcat servlet engine tomcat6
Using CATALINA_BASE: /var/lib/tomcat6
Using CATALINA_HOME: /usr/share/tomcat6
Using CATALINA_TMPDIR: /mnt/tomcat6/tmp
Using JRE_HOME: /usr/lib/jvm/java-6-openjdk
Using CLASSPATH: /usr/share/tomcat6/bin/bootstrap.jar
...fail!Any idea how to debug this?
Comments
Having same problem
EDIT: My comment did not post. Re-trying..
I am also having the same problem, after a update_mercury.sh run. The /var/log/tomcat/catalina.out log shows this error
Error occurred during initialization of VMToo small initial heap
Tomcat was running OK before, so I'm not sure where to look for this. I am running CF server (for FB App) but I've stopped it and Tomcat still won't start.
I am not setting any values in server_tuneables, and allowing the mercury script to set them. free -m (below) shows some free memory, and what I think is enough to start up Tomcat.
free -m
total used free shared buffers cachedMem: 2020 1789 231 0 50 1197
-/+ buffers/cache: 541 1478
Swap: 255 9 246
top
top - 09:08:27 up 9 days, 22 min, 2 users, load average: 0.07, 0.03, 0.00
Tasks: 115 total, 1 running, 113 sleeping, 1 stopped, 0 zombie
Cpu(s): 0.4%us, 0.1%sy, 0.0%ni, 99.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2068848k total, 1870796k used, 198052k free, 51660k buffers
Swap: 262136k total, 9540k used, 252596k free, 1243160k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2089 mysql 20 0 789m 216m 6932 S 0 10.7 485:21.08 mysqld
3567 nobody 20 0 858m 155m 13m S 0 7.7 2:40.93 coldfusion8
2312 nobody 20 0 417m 93m 91m S 0 4.6 14:20.53 varnishd
32519 www-data 20 0 196m 62m 28m S 0 3.1 0:01.53 apache2
2143 root 20 0 69404 44m 2780 S 0 2.2 0:29.13 bcfg2-server
2163 nobody 20 0 74360 30m 836 S 0 1.5 0:31.75 memcached
32522 www-data 20 0 176m 29m 15m S 0 1.5 0:00.80 apache2
32520 www-data 20 0 168m 18m 11m S 3 0.9 0:00.21 apache2
32556 www-data 20 0 168m 17m 11m S 0 0.9 0:00.17 apache2
32557 www-data 20 0 167m 12m 7360 S 0 0.6 0:00.19 apache2
32521 www-data 20 0 166m 11m 7012 S 0 0.6 0:00.40 apache2
32523 www-data 20 0 167m 9.8m 4900 S 0 0.5 0:00.26 apache2
32517 root 20 0 166m 9252 5284 S 0 0.4 0:00.05 apache2
32526 www-data 20 0 166m 5532 1308 S 0 0.3 0:00.00 apache2
32553 www-data 20 0 166m 5528 1308 S 0 0.3 0:00.00 apache2
Would appreciate any pointers.
thanks!
div
I'm getting the same error in
I'm getting the same error in /var/log/tomcat/catalina.out
Error occurred during initialization of VMToo small initial heap
I also have not made any customizations in server_tuneables.
The only non-Pantheon AMI image thing that I have installed/configured is that I installed Webmin and it's dependencies. I don't know if they are even related, but thought I would provide the information here anyway.
Process was as follows:
$ wget http://prdownloads.sourceforge.net/webadmin/webmin_1.520_all.deb$ apt-get install libnet-ssleay-perl libauthen-pam-perl libpam-runtime libio-pty-perl apt-show-versions libapt-pkg-perl
$ dpkg --install webmin_1.520_all.deb
The reason that I mention the webmin install is that previously, the first thing I had done on a server was to run the following:
$ sudo apt-get update$ sudo apt-get upgrade
...and when I ran those, I found that Tomcat6 never started.
Then Greg pointed out here -> http://groups.drupal.org/node/71943#comment-294949
that apt-get updates "should be handled with update_mercury script and it does it in a way that doesn't overwrite pantheon-applied config files (such as those used by tomcat)"
Symetrik Design
Drupal Consulting
http://www.symetrkdesign.com
re: Having same problem
Based on this:
I think that tomcat wants more RAM then the default value is giving it. Try editing the TOMCAT_MEMORY variable in /etc/pantheon/templates/tuneables (default is 96m).
Greg
--
Greg Coit
Systems Administrator
http://www.chapterthree.com
(No subject)
Thanks for the response Greg.
Do you mean /etc/mercury/server_tuneables?
FYI, the comments at the top of /etc/mercury/server_tuneables still say "#Use this file (/root/MERCURY_TUNEABLES) to set variables in config files rather than editing the config file directly."
However, the wiki at http://groups.drupal.org/node/70258 says that it's ok to edit them in /etc/mercury/server_tuneables
I upped the memory for TOMCAT_MEMORY to 128, 256, and 512 with the same result.
I stopped BCFG2 server and made sure there were no process running.
Edited /etc/mercury/server_tuneables
$ sudo /etc/init.d/bcfg2-server start (double checked that the service started)
$ sudo bcfg2 -vqed
I'm still seeing this in /var/log/tomcat6/catalina.out
Error occurred during initialization of VMToo small initial heap
The following is the output from sudo bcfg2 -vqed
ubuntu@dev:/mnt/tomcat6$ sudo bcfg2 -vqed
{'profile': False, 'file': False, 'filelog': None, 'dryrun': False, 'verbose': True, 'extra': True, 'ca': '/etc/bcfg2.crt', 'args': [], 'remove': False, 'bundle': [], 'quick': True, 'user': 'root', 'key': '/etc/bcfg2.key', 'setup': '/etc/bcfg2.conf', 'password': 'F24XxRlh', 'help': False, 'retries': '3', 'omit-lock-check': False, 'certificate': '/etc/bcfg2.crt', 'paranoid': False, 'indep': False, 'decision': False, 'cache': False, 'serverCN': None, 'servicemode': 'default', 'encoding': 'ascii', 'decision-list': False, 'kevlar': False, 'debug': True, 'server': 'https://localhost:6789', 'drivers': ['Action', 'APT', 'Blast', 'Chkconfig', 'DebInit', 'Encap', 'IPS', 'FreeBSDInit', 'FreeBSDPackage', 'launchd', 'Portage', 'POSIX', 'RPMng', 'RcUpdate', 'SMF', 'SYSV', 'YUMng'], 'interactive': False}
Running probe php_memory
Probe php_memory has result:
256
Running probe memcached_memory
Probe memcached_memory has result:
128
Running probe mysql_max_connections
Probe mysql_max_connections has result:
23
Running probe innodb_buffer_pool_size
Probe innodb_buffer_pool_size has result:
256
Running probe innodb_log_file_size
Probe innodb_log_file_size has result:
1073741824
Running probe tomcat_max_threads
Probe tomcat_max_threads has result:
100
Running probe varnish_vcl_hash
Probe varnish_vcl_hash has result:
Running probe varnish_vcl_fetch
Probe varnish_vcl_fetch has result:
Running probe varnish_vcl_error
Probe varnish_vcl_error has result:
Running probe apparmor_mysql
Probe apparmor_mysql has result:
/mnt/mysql/** rwkl,
Running probe apc_memory
Probe apc_memory has result:
96
Running probe groups
Probe groups has result:
-e
group:deb
group:lucid
group:ubuntu
group:i386
Running probe apache_maxclients
Probe apache_maxclients has result:
18
Running probe varnish_memory
Probe varnish_memory has result:
128
Running probe key_buffer_size
Probe key_buffer_size has result:
8
Running probe varnish_vcl_recv
Probe varnish_vcl_recv has result:
Running probe tomcat_memory
Probe tomcat_memory has result:
512
Loaded tool drivers:
APT Action DebInit POSIX
The following entries are included multiple times:
Package:libpng12-0 Package:psmisc Package:python-configobj Service:postfix
Package:make Package:python-apt Package:python-support
The following packages are specified in bcfg2:
adduser libdrm-nouveau1 memcached
alien libdrm-radeon1 memtest86+
apache2 libdrm2 mime-support
apache2-mpm-prefork libecj-java mlocate
apache2-utils libecj-java-gcj module-init-tools
apache2.2-bin libedit2 mount
apparmor libept0 mountall
apparmor-utils libfile-copy-recursive-perl mtr-tiny
apport libfile-slurp-perl mysql-client
apport-symptoms libflac8 mysql-client-5.1
apt libfont-afm-perl mysql-client-core-5.1
apt-transport-https libfontconfig1 mysql-common
apt-utils libfribidi0 mysql-server
aptitude libgamin0 mysql-server-5.1
at libgcc1 mysql-server-core-5.1
autoconf libgcj-bc nano
autoconf2.13 libgcj-common ncurses-base
automake1.4 libgcj9-0 ncurses-bin
base-files libgcj9-jar net-tools
base-passwd libgcrypt11 netbase
bash libgd2-xpm netcat-openbsd
bash-completion libgdbm3 ntfs-3g
bcfg2 libgeoip1 ntpdate
bcfg2-server libgif4 openjdk-6-jre-lib
bind9-host libglib2.0-0 openssh-client
binutils libgmp3c2 openssh-server
bsdmainutils libgnutls26 openssl
bsdutils libgpg-error0 os-prober
busybox-initramfs libgpm2 parted
busybox-static libgssapi-krb5-2 passwd
byobu libgtk2.0-0 patch
bzip2 libgtk2.0-common pciutils
bzr libhtml-format-perl perl
ca-certificates libidn11 perl-base
ca-certificates-java libio-string-perl perl-modules
cloud-init libisc60 php-apc
cloud-utils libisccc60 php-pear
command-not-found libisccfg60 php5
command-not-found-data libjs-jquery php5-cli
console-setup libk5crypto3 php5-curl
console-terminus libkeyutils1 php5-dev
consolekit libklibc php5-gd
coreutils libkrb5-3 php5-imagick
cpio libkrb5support0 php5-memcache
cpp liblcms1 php5-mysql
cpp-4.4 libldap-2.4-2 php5-xmlrpc
cpu-checker liblist-moreutils-perl plymouth
cron liblocale-gettext-perl plymouth-theme-ubuntu-text
curl liblockfile1 pm-utils
cvs libltdl7 popularity-contest
dash liblwres60 portmap
dbus libmagic1 postfix
debconf libmailtools-perl powermgmt-base
debconf-i18n libmoose-perl ppp
debhelper libmpfr1ldbl pppconfig
debianutils libmro-compat-perl pppoeconf
debsums libncurses5 procps
default-jre libncursesw5 psfontmgr
default-jre-headless libnet-daemon-perl psmisc
defoma libnewt0.52 psmisc
dhcp3-client libnih-dbus1 python
dhcp3-common libnih1 python-apport
diff libnspr4-0d python-apt
diffutils libnss3-1d python-apt
dmidecode libogg0 python-boto
dmsetup libpam-ck-connector python-central
dnsutils libpam-modules python-cheetah
dosfstools libpam-runtime python-configobj
dpkg libpam0g python-configobj
dpkg-dev libpango1.0-0 python-dbus
e2fslibs libpango1.0-common python-fam
e2fsprogs libpaper1 python-gamin
ec2-ami-tools libparams-util-perl python-gdbm
ec2-api-tools libparams-validate-perl python-genshi
ec2-consistent-snapshot libparse-debianchangelog-perl python-gnupginterface
ed libpcre3 python-gobject
eject libpixman-1-0 python-httplib2
ethtool libplrpc-perl python-launchpadlib
euca2ools libplymouth2 python-lazr.restfulclient
file libpng12-0 python-lazr.uri
findutils libpng12-0 python-lxml
fontconfig libpopt0 python-m2crypto
fontconfig-config libpulse0 python-minimal
friendly-recovery libpython2.6 python-newt
ftp libreadline6 python-oauth
fuse-utils librpc-xml-perl python-openssl
gamin libruby1.8 python-pam
gcc-4.4-base libsasl2-2 python-pexpect
gcj-4.3-base libsasl2-modules python-pkg-resources
geoip-database libscope-guard-perl python-problem-report
gettext-base libselinux1 python-pycurl
ghostscript libsepol1 python-serial
gnupg libservlet2.5-java python-simplejson
gnupg-curl libsigc++-2.0-0c2a python-smartpm
gpgv libslang2 python-software-properties
grep libsndfile1 python-support
groff-base libsqlite3-0 python-support
grub-common libss2 python-twisted-bin
grub-pc libssl-dev python-twisted-core
gzip libssl0.9.8 python-twisted-web
hal libstdc++6 python-wadllib
hal-info libsub-exporter-perl python-yaml
hdparm libsub-install-perl python-zope.interface
hostname libsub-name-perl python2.6
icedtea-6-jre-cacao libsub-uplevel-perl python2.6-minimal
ifupdown libt1-5 radeontool
info libtask-weaken-perl readline-common
initramfs-tools libtasn1-3 rhino
initramfs-tools-bin libterm-readkey-perl rpm
initscripts libtest-exception-perl rsync
insserv libtext-charwidth-perl rsyslog
install-info libtext-iconv-perl runurl
iproute libtext-wrapi18n-perl screen
iptables libthai-data sed
iputils-arping libthai0 sensible-utils
iputils-ping libtimedate-perl sgml-base
iputils-tracepath libtomcat6-java shared-mime-info
irqbalance libtool shtool
iso-codes libtry-tiny-perl smartdimmer
java-common libts-0.0-0 snmp
kbd libudev0 snmpd
klibc-utils libusb-0.1-4 ssh-import
landscape-client libuuid1 ssl-cert
landscape-common libvorbis0a strace
language-selector-common libvorbisenc2 subversion
laptop-detect libwrap0 sudo
less libx11-6 sysv-rc
libaccess-bridge-java libx11-data sysvinit-utils
libaccess-bridge-java-jni libxapian15 tar
libacl1 libxau6 tasksel
libalgorithm-c3-perl libxcb-render-util0 tasksel-data
libapache2-mod-php5 libxcb-render0 tcpd
libapache2-mod-rpaf libxcb1 tcpdump
libapparmor-perl libxcomposite1 telnet
libapparmor1 libxcursor1 time
libaprutil1-dbd-sqlite3 libxdamage1 tomcat6
libaprutil1-ldap libxdmcp6 tomcat6-common
libatk1.0-0 libxext6 tsconf
libatm1 libxfixes3 ttf-dejavu-core
libattr1 libxft2 tzdata
libavahi-client3 libxi6 tzdata-java
libavahi-common-data libxinerama1 ubuntu-keyring
libavahi-common3 libxml-libxml-perl ubuntu-minimal
libbind9-60 libxml-sax-expat-perl ubuntu-serverguide
libblkid1 libxml-simple-perl ubuntu-standard
libbsd0 libxml2 ucf
libbz2-1.0 libxml2-utils udev
libc-bin libxmuu1 ufw
libc-dev-bin libxpm4 unattended-upgrades
libc6 libxrandr2 unzip
libc6-dev libxrender1 update-inetd
libc6-xen libxslt1.1 update-manager-core
libcairo2 libxtst6 update-motd
libcap2 linux-ec2 update-notifier-common
libclass-accessor-perl linux-firmware upstart
libclass-c3-perl linux-image-2.6.32-21-virtual ureadahead
libclass-mop-perl linux-image-2.6.32-22-virtual usbutils
libcomerr2 linux-image-2.6.32-305-ec2 util-linux
libcommons-collections-java linux-image-ec2 uuid-runtime
libcommons-dbcp-java linux-image-virtual varnish
libcommons-pool-java linux-libc-dev vbetool
libcups2 linux-virtual vim
libcurl3 locales vim-common
libcurl3-gnutls lockfile-progs vim-runtime
libcwidget3 login vim-tiny
libdata-optlist-perl logrotate vnstat
libdatrie1 lsb-base w3m
libdb4.7 lsb-release wget
libdb4.8 lshw whiptail
libdbd-mysql-perl lsof wireless-tools
libdbi-perl ltrace wpasupplicant
libdbus-1-3 lzma x11-common
libdevel-globaldestruction-perl m4 xauth
libdevmapper1.02.1 make xfsprogs
libdigest-hmac-perl make xkb-data
libdigest-sha1-perl makedev xml-core
libdirectfb-1.2-0 man-db xz-utils
libdns64 manpages zlib1g
libdrm-intel1 mawk zlib1g-dev
The following packages are prereqs added by Packages:
apache2.2-common libhal-storage1 libsnmp-base linux-image-2.6.32-308-ec2
libcupsimage2 libhal1 libsnmp15 openjdk-6-jre-headless
libexpat1 libhtml-parser-perl libtiff4 php5-common
libfreetype6 libmysqlclient16 libwww-perl
libgs8 libparted0debian1 linux-image-2.6.32-24-virtual
Path /mnt/tomcat6/tmp group wrong
Path /mnt/tomcat6/tmp permissions are 0755 should be 1777
Phase: initial
Correct entries: 619
Incorrect entries: 2
Total managed entries: 621
Unmanaged entries: 79
Installing Directory /mnt/tomcat6/tmp
Found a pre-existing directory at /mnt/tomcat6/tmp
Installing ConfigFile /etc/default/tomcat6
The Following Bundles have been modified:
amazon-web-services tomcat6
Restarting service dns-clean
> /usr/sbin/invoke-rc.d dns-clean restart
Stopping service grub-common
> /usr/sbin/invoke-rc.d grub-common stop
Restarting service landscape-client
> /usr/sbin/invoke-rc.d landscape-client restart
< * Stopping landscape-client daemon
< ...fail!
< landscape-client is not configured, please run landscape-config.
Restarting service pppd-dns
> /usr/sbin/invoke-rc.d pppd-dns restart
Restarting service rsync
> /usr/sbin/invoke-rc.d rsync restart
Restarting service tomcat6
> /usr/sbin/invoke-rc.d tomcat6 restart
< * Starting Tomcat servlet engine tomcat6
< ...fail!
< invoke-rc.d: initscript tomcat6, action "restart" failed.
Failed to manipulate service tomcat6
Running Action clear_ib_logfiles
> bash /var/lib/bcfg2/Probes/innodb_log_file_size
< 1073741824
Action: bash /var/lib/bcfg2/Probes/innodb_log_file_size got rc 0
Running Action vnstat_db_eth0
> vnstat -u -i eth0
Action: vnstat -u -i eth0 got rc 0
Running Action restart_apparmor
> service apparmor restart
< * Reloading AppArmor profiles
< ...done.
Action: service apparmor restart got rc 0
Phase: final
Correct entries: 621
Incorrect entries: 0
Total managed entries: 621
Unmanaged entries: 79
Package:apt-show-versions Package:libgc1c2 Package:librpm0
Package:automake Package:libgcj10 Package:librpmbuild0
Package:autotools-dev Package:libgomp1 Package:librpmio0
Package:bc Package:libhtml-tagset-perl Package:libsensors4
Package:gawk Package:libhtml-tree-perl Package:libsm6
Package:gcc Package:libice6 Package:libsvn1
Package:gcc-4.4 Package:libio-pty-perl Package:libsysfs2
Package:gcj-4.4-base Package:libiw30 Package:liburi-perl
Package:gettext Package:libjasper1 Package:libvarnish1
Package:gsfonts Package:libjline-java Package:libx86-1
Package:html2text Package:libjpeg62 Package:libxml-namespacesupport-perl
Package:intltool-debian Package:liblua5.1-0 Package:libxml-parser-perl
Package:libapr1 Package:liblzma1 Package:libxml-sax-perl
Package:libaprutil1 Package:libmagickcore2 Package:libxt6
Package:libapt-pkg-perl Package:libmagickwand2 Package:libyaml-0-2
Package:libasound2 Package:libneon27-gnutls Package:linux-image-2.6.32-25-virtual
Package:libauthen-pam-perl Package:libnet-ssleay-perl Package:openjdk-6-jre
Package:libc6-i686 Package:libnl1 Package:po-debconf
Package:libcap-ng0 Package:libntfs-3g75 Package:rpm-common
Package:libck-connector0 Package:libopenssl-ruby Package:rpm2cpio
Package:libcroco3 Package:libopenssl-ruby1.8 Package:ruby
Package:libdbus-glib-1-2 Package:libpcap0.8 Package:ruby1.8
Package:libeggdbus-1-0 Package:libpci3 Package:webmin
Package:libelf1 Package:libpciaccess0 Service:webmin
Package:libevent-1.4-2 Package:libpcsclite1 Service:webmin_1
Package:libffi5 Package:libperl5.10
Package:libfuse2 Package:libpolkit-gobject-1-0
Symetrik Design
Drupal Consulting
http://www.symetrkdesign.com
same here.
Same here. I tried setting tomcat memory to 128, ,256 but still won't start. I stopped all other processes (varnish, memcache, apache, mysql, colfdfusion, fail2ban) and it still would not start.
I do have monit and munin running which I did not mention before.
I believe I have done a few apt-get upgrades outside of the update_mercury.sh process. I thought I read I should allow the update_mercury.sh script to handle the upgrades, so perhaps this is part of the problem?
Thanks!
div
Can you check
Can you check /etc/default/tomcat6
Specifically, the JAVA_OPTS argument should look something like:
JAVA_OPTS="-Djava.awt.headless=true -Xmx96m"
There should be an "m" at the end of -Xmx96m (or 128m/256m/etc). At one point I remember this was missing, and it defaulted to Kb.
That did it for me too. I
That did it for me too. I actually thought about that when originally editing /etc/mercury/server_tuneables, but left it as just 128.
Simply setting this as 96m, which would be the default, worked for me.
Thank you fine sir!
Symetrik Design
Drupal Consulting
http://www.symetrkdesign.com
Worked
This worked for me too, I guess 128kb was a little low :-).
Thank you
That was it
that worked for me. The "m" was missing on the end of the line. adding and tomcat started right up.
thanks for the pointer!
div
Should we add M to all the
Should we add M to all the settings in Tuneables? Can you share your tuneable settings here? Im using 1.5 g memory linode and trying to get things fast...
www.thecenterofthenet.com
http://cn.linkedin.com/in/roberpteatonjr
http://www.facebook.com/robertp.eaton
http://www.alamy.com/stock-photography/6A42F1A1-E40A-48FD-8C7F-FF258A401...