It’s not a secret that there are some memory leaks in our CIM Server and it looks like not only XenServer has this kind of problems. Recetly I spent some time helping my colleague to troubleshooting problem like this and decided to share below procedures


Procedure 1: Using core files.

  1. Set core file destination and pattern:
    sysctl -w kernel.core_pattern="/var/tmp/core.%e.%h.%p"
    sysctl -q kernel.core_pattern 
    

    Above command sets core file patter is as follow:

    /var/tmp/core.<PROCESSNAME>.<HOSTNAME>.<PID>

  2. Edit <strong><em>/etc/init.d/openpegasus</em></strong> file and add <strong><em>ulimit -c unlimited</em></strong> line before cimserver starts.
    Or apply below patch on <strong><em>/etc/init.d/openpegasus</em></strong>
    *** /etc/init.d/openpegasus.00  2013-04-05 14:39:48.000000000 +0530
    --- /etc/init.d/openpegasus  2013-04-05 14:31:42.000000000 +0530
    *************** start() {
    *** 11,16 ****
    --- 11,17 ----
      
      stop() {
        echo -n "Stopping OpenPegasus... "
    +   ulimit -c unlimited
              /opt/openpegasus/bin/cimserver -s
              echo "done."
      }
    

    To apply this patch:

    • save it to text file in XenServer dom0, ie: <strong><em>/root/cim.patch</em></strong>
    • run: <strong><em>patch &lt; /root/cim.patch</em></strong>
  3. Restart cimserver
    service openpegasus restart
    
  4. When cimserver starts consume abnormal amount of memory kill cimserver with SIGABRT (6) signal.
    kill -SIGABRT $(pgrep cimserver)
    
  5. Check /var/tmp/ for new core file.


Procedure 2: Using valgrind framework.

  1. Stop cimserver watchdog
    touch /opt/xs-cim-cmpi/watchdog-off
    
  2. Set cimserver to run in foreground:
    /opt/openpegasus/bin/cimconfig -s daemon=false -p 
  3. Start cimserver using valgrind
    valgrind --leak-check=full --log-file=/root/cimserver.valgrind%p.log --trace-children=yes /opt/openpegasus/bin/cimserver
  4. When issue occures stop valgrind and collect log file from /var/tmp
    Follow rest of the steps to revert cimconfig original configuration.
  5. Set cimserver to run in background:
    /opt/openpegasus/bin/cimconfig -s daemon=true -p 
  6. Enable watchdog:
    rm -vf /opt/xs-cim-cmpi/watchdog-off