Jul 29 11:15:09 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “heartbeat approaching timeout”. host[0].time_since_last_hb=39543.
Jul 29 11:15:09 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “State-File approaching timeout”. host[0].time_since_last_update_on_sf=37508.
Jul 29 11:15:11 JST 2009 [debug] SM: SF domain is updated [sfdomain = (_1)].
Jul 29 11:15:11 JST 2009 [debug] FH: Start fault handler.
Jul 29 11:15:12 JST 2009 [debug] FH: waiting for HB from host (0), time since last HB receive = 42535.
Jul 29 11:15:14 JST 2009 [debug] HB: HB domain is updated [hbdomain = (_m)].
Jul 29 11:15:14 JST 2009 [debug] FH: HB/SF state has become stable.
Jul 29 11:15:14 JST 2009 [warn] All hosts now have a consistent view.
Jul 29 11:15:14 JST 2009 [warn]     HB domain = (01)
Jul 29 11:15:14 JST 2009 [warn]     SF domain = (01)
Jul 29 11:15:14 JST 2009 [debug] FH: All hosts now have consistent view to the pool membership.
Jul 29 11:15:15 JST 2009 [debug] FH: I have won.
Jul 29 11:15:15 JST 2009 [debug] SM: Node (0) will be removed from liveset.
Jul 29 11:15:35 JST 2009 [notice] Liveset has been updated.  new liveset = (01)
Jul 29 11:15:35 JST 2009 [debug] FH: End fault handler.
Jul 29 11:15:49 JST 2009 [info] SC: (script_service_do_query_liveset) “Heartbeat approaching timeout” turned FALSE
Jul 29 11:15:49 JST 2009 [info] SC: (script_service_do_query_liveset) “State-file approaching timeout” turned FALSE
Jul 29 11:19:48 JST 2009 [debug] HB: HB domain is updated [hbdomain = (@m)].
Jul 29 11:19:59 JST 2009 [debug] SM: SF domain is updated [sfdomain = (@1)].
Jul 29 11:19:59 JST 2009 [debug] Join Agent: Send ack to join request from host (0).
Jul 29 11:19:59 JST 2009 [info] Join Agent: Join request from host (0) is accepted by the local host.
Jul 29 11:19:59 JST 2009 [notice] Liveset has been updated.  new liveset = (11)
Jul 29 11:19:59 JST 2009 [info] Join Agent: proposed_liveset = (11)
Jul 29 11:28:25 JST 2009 [debug] WD: now – settime = 27872 timeout = 55000.
Jul 29 11:28:25 JST 2009 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Jul 29 11:28:30 JST 2009 [debug] WD: now – settime = 32881 timeout = 55000.
Jul 29 11:28:30 JST 2009 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Jul 29 11:28:31 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “heartbeat approaching timeout”. host[0].time_since_last_hb=37448.
Jul 29 11:28:31 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “State-File approaching timeout”. host[0].time_since_last_update_on_sf=38708.
Jul 29 11:28:31 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “State-File approaching timeout”. host[1].time_since_last_update_on_sf=38708.
Jul 29 11:28:33 JST 2009 [debug] SM: SF domain is updated [sfdomain = (__)].
Jul 29 11:28:33 JST 2009 [debug] FH: Start fault handler.
Jul 29 11:28:33 JST 2009 [debug] FH: waiting for HB from host (0), time since last HB receive = 39230.
Jul 29 11:28:35 JST 2009 [debug] WD: now – settime = 37882 timeout = 55000.
Jul 29 11:28:35 JST 2009 [info] WD: watchdog has not been updated at least for half timeout id=1 label=statefile.
Jul 29 11:28:35 JST 2009 [debug] FH: HB/SF state has become stable.
Jul 29 11:28:35 JST 2009 [debug] HB: HB domain is updated [hbdomain = (_m)].
Jul 29 11:28:36 JST 2009 [warn] All hosts now have a consistent view.
Jul 29 11:28:36 JST 2009 [warn]     HB domain = (01)
Jul 29 11:28:36 JST 2009 [warn]     SF domain = (00)
Jul 29 11:28:36 JST 2009 [debug] FH: All hosts now have consistent view to the pool membership.
Jul 29 11:28:36 JST 2009 [warn] SM: partition_size[0] = 0
Jul 29 11:28:36 JST 2009 [warn] SM: partition_size[1] = 0
Jul 29 11:28:36 JST 2009 [warn] SM: winner_index = -1
Jul 29 11:28:36 JST 2009 [debug] FH: I have lost.
Jul 29 11:28:36 JST 2009 [err] Survival rule failed (1905) FH: Survival Rule is not met for the local host.  – Self-Fence.
Jul 29 11:28:36 JST 2009 [info] watchdog_selffence.
Aug 03 21:22:02 JST 2009 [notice] HA daemon started – built at Jan 13 05:27:02 EST 2009 – 8b0cfd15d7fa
Aug 03 21:22:02 JST 2009 [info] CONF: my_index=1 num_host=2 t1=5 T1=40 t2=5 T2=40 Wh=40 Ws=55 Tboot=100 Tenable=100 tXapi=60 TXapi=120 RestartXapi=1 TRestartXapi=300 Tlicence=30
Aug 03 21:22:02 JST 2009 [info] LOG: logmask = 100000
Aug 03 21:22:02 JST 2009 [info] LOG:  OFF:18(DUMPPACKET)
Aug 03 21:22:02 JST 2009 [info] LOG:  OFF:19(TRACE)
Aug 03 21:22:02 JST 2009 [info] LOG:  ON :20(FH_TRACE)
Aug 03 21:22:02 JST 2009 [info] LOG:  OFF:21(LM_TRACE)
Aug 03 21:22:02 JST 2009 [info] LOG:  OFF:22(SCRIPT)
Aug 03 21:22:02 JST 2009 [info] LOG:  OFF:23(SC_WARNING)
Aug 03 21:22:02 JST 2009 [info] COM: com_initialize(0).
Aug 03 21:22:02 JST 2009 [info] SF: phase 0 initialization…
Aug 03 21:22:02 JST 2009 [info] LM: lm_initialize(0).
Aug 03 21:22:02 JST 2009 [info] BM: bm_initialize(0).
Aug 03 21:22:02 JST 2009 [info] HB: hb_initialize(0).
Aug 03 21:22:02 JST 2009 [info] Xapimon: xapimon_initialize(0).
Aug 03 21:22:02 JST 2009 [info] SC: script_initialize(0).
Aug 03 21:22:02 JST 2009 [info] SM: sm_initialize(0).
Aug 03 21:22:02 JST 2009 [info] COM: com_initialize(1).
Aug 03 21:22:02 JST 2009 [info] SF: phase 1 initialization…
Aug 03 21:22:02 JST 2009 [info] WD: watchdog mode = HYPERVISOR.
Aug 03 21:22:02 JST 2009 [info] WD: (watchdog_create) success label=statefile id=1.
Aug 03 21:22:02 JST 2009 [debug] WD: watchdog id file /var/run/xhad.wd.id is updated.
Aug 03 21:22:02 JST 2009 [info] LM: lm_initialize(1).
Aug 03 21:22:02 JST 2009 [info] BM: bm_initialize(1).
Aug 03 21:22:02 JST 2009 [info] HB: hb_initialize(1).
Aug 03 21:22:02 JST 2009 [info] WD: (watchdog_create) success label=heartbeat id=2.
Aug 03 21:22:02 JST 2009 [debug] WD: watchdog id file /var/run/xhad.wd.id is updated.
Aug 03 21:22:02 JST 2009 [info] Xapimon: xapimon_initialize(1).
Aug 03 21:22:02 JST 2009 [info] SC: script_initialize(1).
Aug 03 21:22:02 JST 2009 [info] SM: sm_initialize(1).
Aug 03 21:22:02 JST 2009 [info] State File read/write enabled.
Aug 03 21:22:02 JST 2009 [info] Heartbeat send/receive enabled.
Aug 03 21:22:05 JST 2009 [debug] HB: HB domain is updated [hbdomain = (@m)].
Aug 03 21:22:05 JST 2009 [debug] SM: heartbeat is ready
Aug 03 21:22:08 JST 2009 [notice] HA daemon successfuly started acceesing the State-File.
Aug 03 21:22:12 JST 2009 [debug] BM: MII Status: down (interface  eth1)
Aug 03 21:22:12 JST 2009 [warn] BM: bonding status has changed to DEGRADED.
Aug 03 21:22:13 JST 2009 [debug] SM: SF domain is updated [sfdomain = (@0)].
Aug 03 21:22:18 JST 2009 [debug] SM: SF domain is updated [sfdomain = (1@)].
Aug 03 21:22:18 JST 2009 [debug] SM: statefile is ready
Aug 03 21:22:18 JST 2009 [debug] SM: other hosts are ready
Aug 03 21:22:18 JST 2009 [info] Start Criteria: pool state=ACTIVE; excluded=FALSE; liveset=EXIST
Aug 03 21:22:18 JST 2009 [info] Start Criteria: Joining the existing liveset.
Aug 03 21:22:18 JST 2009 [info] Start Criteria: current_liveset = (10)
Aug 03 21:22:18 JST 2009 [info] Start Criteria: proposed_liveset = (10)
Aug 03 21:22:18 JST 2009 [info] Start Criteria has been met for the local host.
Aug 03 21:22:18 JST 2009 [debug] Join: trying to join [proposed_liveset = (11)].
Aug 03 21:22:22 JST 2009 [notice] BM: bonding status has changed to GOOD.
Aug 03 21:22:23 JST 2009 [info] Join: joined successfully [new liveset = (11)].
Aug 03 21:22:23 JST 2009 [info] Joined.
Aug 03 21:22:23 JST 2009 [info] Xapi monitor started.
Aug 03 21:22:23 JST 2009 [info] Fencing is armed.
Aug 03 21:22:23 JST 2009 [notice] The local host has transitioned to online state.
Aug 03 21:22:23 JST 2009 [debug] HA daemon set scheduler policy=2 priority=50
Aug 03 21:22:23 JST 2009 [debug] HA daemon mlockall succeeded
Aug 03 21:22:24 JST 2009 [info] SC: propose_master enter.
Aug 03 21:22:24 JST 2009 [notice] propose_master returns FALSE.
Aug 03 21:22:24 JST 2009 [info] SC: propose_master enter.
Aug 03 21:22:24 JST 2009 [notice] propose_master returns FALSE.
Aug 07 03:50:41 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “heartbeat approaching timeout”. host[0].time_since_last_hb=21155.
Aug 07 03:51:01 JST 2009 [warn] SC: (script_service_do_query_liveset) reporting “State-File approaching timeout”. host[0].time_since_last_update_on_sf=38056.
Aug 07 03:51:03 JST 2009 [debug] SM: SF domain is updated [sfdomain = (_1)].
Aug 07 03:51:03 JST 2009 [debug] FH: Start fault handler.
Aug 07 03:51:03 JST 2009 [debug] FH: waiting for HB from host (0), time since last HB receive = 43580.
Aug 07 03:51:04 JST 2009 [debug] HB: HB domain is updated [hbdomain = (_m)].
Aug 07 03:51:04 JST 2009 [debug] FH: HB/SF state has become stable.
Aug 07 03:51:05 JST 2009 [warn] All hosts now have a consistent view.
Aug 07 03:51:05 JST 2009 [warn]     HB domain = (01)
Aug 07 03:51:05 JST 2009 [warn]     SF domain = (01)
Aug 07 03:51:05 JST 2009 [debug] FH: All hosts now have consistent view to the pool membership.
Aug 07 03:51:05 JST 2009 [debug] FH: I have won.
Aug 07 03:51:05 JST 2009 [debug] SM: Node (0) will be removed from liveset.
Aug 07 03:51:25 JST 2009 [notice] Liveset has been updated.  new liveset = (01)
Aug 07 03:51:25 JST 2009 [debug] FH: End fault handler.
Aug 07 03:51:41 JST 2009 [info] SC: (script_service_do_query_liveset) “Heartbeat approaching timeout” turned FALSE
Aug 07 03:51:41 JST 2009 [info] SC: (script_service_do_query_liveset) “State-file approaching timeout” turned FALSE
Aug 07 03:51:41 JST 2009 [info] SC: propose_master enter.
Aug 07 03:51:42 JST 2009 [notice] propose_master returns TRUE.
Aug 07 03:51:53 JST 2009 [info] SC: propose_master enter.
Aug 07 03:51:53 JST 2009 [notice] propose_master returns TRUE.