14:59 seanhandleyWe're trying to get foreman to tell nagios to shut up automatically with hooks during machine rebuilds. So far it goes into scheduled downtime mode in nagios when the machine is put into build in foreman. The issue is telling nagios to pay attention again. If we do it on the before_provision hook then some services which nagios monitors won't be up as puppet won't have configured them. Any suggestions?

15:02 coffmant1 has joined (~coffmant@fw1-wireless.snapbs.com)

15:04 wrath0r has joined (~wrath0r@38.97.106.132)

15:04 coffmant has left IRC (Ping timeout: 240 seconds)

15:06 mmello has joined (~Marcelo@cpe-107-015-251-245.nc.res.rr.com)

15:06 sloranz has joined (~sloranz@redhat/jboss/sloranz)

15:07 valeri_ufoohadlevy: you there? :)

15:07 ybronhei has left IRC (Ping timeout: 240 seconds)

15:08 neha_ has left IRC (Ping timeout: 264 seconds)

15:11 gwmngilfen|lunch is now known as gwmngilfen

15:12 gwmngilfenvaleri_ufo: it should be part of our puppet-foreman module, iirc

15:12 suresh12 has joined (~suresh12@122.174.101.119)

15:13 kscherer has joined (~kscherer@128.224.252.2)

15:15 eLobato|lunchseanhandley: unfortunately not, it can be easily added though

15:15 eLobato|lunchseanhandley: there is after_build if that helps you

15:15 gwmngilfenseanhandley: also you could run puppet --tags nagios in the %post before rebooting

15:15 seanhandleyeLobato|lunch: I'm using after_build already - that's the hook that tells nagios to ignore the host for a while

15:15 eLobato|lunch is now known as eLobato

15:15 gwmngilfenor whatever you need to get things set back up

15:16 valeri_ufogwmngilfen: and puppet-foreman should be installed, since i used the foreman installer to bootstrap foreman, right?

15:16 seanhandleyI think an after_provision hook would be ideal really

15:16 gwmngilfenvaleri_ufo: right, but the default install of the installer package isn;t in your puppet module path

15:16 YamakasYis it possible to call a specific manifext at the end of a build, in the finish script ?

15:16 YamakasYI need to run this one as first ever

15:16 gwmngilfenvaleri_ufo: unless you moved it

15:17 YamakasYso after the build would be best I think

15:17 valeri_ufogwmngilfen: find says: /usr/share/foreman-installer/modules/foreman/lib/puppet/parser/functions/foreman.rb

15:17 seanhandleyeLobato: gwmngilfen: If I'm understanding this correctly...

15:17 seanhandleyhttps://github.com/theforeman/foreman/blob/develop/app/models/host/managed.rb#L27

15:18 valeri_ufogwmngilfen: is this the right one or is this just for the installer itself and i need a different parser?

15:18 seanhandleyI can remove the :only => :before directive there

15:18 eLobatoYamakasY: copy the manifest to the host and run 'puppet apply xxx.pp' on your finish script?

15:18 gwmngilfenvaleri_ufo: thats the one I think, you can probably just copy the module into your modulepath

15:19 valeri_ufogwmngilfen: but shouldn't it work out of the box, i thought i could use the foreman function since i provisioned all hosts by foreman and use foreman as enc etc

15:20 gwmngilfenvaleri_ufo: for puppet to use a function, it must be somewhere in your modulepath, and by default /usr/share/foreman-installer/modules is not in puppet.conf

15:20 eLobatoseanhandley: how are you provisioning your machine and at which point you want nagios to come back up?

15:20 seanhandleywith puppet

15:21 seanhandleywe're sending an API call to nagios that schedules a downtime period for the host

15:21 YamakasYeLobato: mhh that might be an idea

15:21 valeri_ufogwmngilfen: i understand, i thouht foreman exported those functions or modified the path

15:21 seanhandleyeLobato: so once foreman registers that puppet has provisioned the host then it's safe to run the hook and tell nagios to start paying attention again

15:22 seanhandleyso really, if I'm reading the code correctly, it's a case of allowing an :after magic method on the :provision call

15:22 YamakasYeLobato: need to check as the finisch script also runs puppet once I thought

15:22 chiemseesurfer has left IRC (Quit: http://quassel-irc.org - Chat comfortably. Anywhere.)

15:22 gwmngilfenseanhandley: so you want to wait until the first report is in after the reboot? what if it has errors in it?

15:22 seanhandleygwmngilfen: Then so be it - as far as Foreman's concerned the host has been rebuilt and provisioned

15:23 seanhandleyif there are problems then having nagios send alerts isn't a problem

15:23 gwmngilfenok, rephrase - you want at least one puppet run? or are you wanting nagios back online *at* the reboot point (before it has come back up into the OS

15:23 seanhandleywe're doing this because during rebuild of a machine we get a lot of meaningless alerts (the machine's being rebuilt, of course the MySQL service isn't responding) etc

15:23 eLobatoseanhandley: you're doing this through a plugin? I don't quite follow. you can query Foreman or PuppetDB in your Puppet manifests to check for whatever you're looking for in the report

15:24 seanhandleywe're using the foreman_hooks plugin

15:24 seanhandleyit's running a custom script

15:24 seanhandleythat script makes an API call to nagios

15:24 seanhandleyand says "Hey, ignore this host for a while"

15:24 seanhandleythat part works *perfectly*

15:24 seanhandleyand it uses the after_build hook

15:24 straldi1eLobato thanks for answering, but the simple things should have in foreman a combobox like the virt-manager one. I use a kickstart pxe, and aI do not understand how the foreman_bootdisk could help me.

15:24 seanhandleywhich is triggered after a host is put into build mode

15:25 seanhandleyI have another script currently in the before_provision dir in the hooks

15:25 seanhandleyand that runs and says "Nagios, pay attention to this host again"

15:25 robertapi has left IRC (Quit: Page closed)

15:25 gwmngilfenseanhandley: i ask because once you go past that reboot point, foreman has little further interaction with the host. i *think* there is a before_provision hook, which confusingly fires at the end of the build provess

15:25 gwmngilfen(just as confusing as after_build being before the build actually happens :P)

15:25 seanhandleythe problem is that before_provision is too soon - we need provision to finish execution so that the services nagios monitors have a chance to start again

15:26 straldi1foreman has I said have set before network and after disk, but the kvm master has different setting so I have to change it form the kvm master in this way.

15:26 jfarschman has joined (~Adium@205.169.68.218)

15:26 jfarschman is now known as MilesDenver

15:26 seanhandleyyeah the naming is weird gwmngilfen :-)

15:26 eLobatostraldi1: agreed, we do have that combobox for physical hosts in fact, foreman_bootdisk could help you to create a disk so that when your hosts boot from disk by mistake, they still are able to connect to foreman

15:26 gwmngilfenseanhandley: so the issue is that I thing before_provision is the last place formen_hooks can fire

15:26 seanhandleyso what you're saying is that before_provision is fired *after* puppet has finished ?

15:26 gwmngilfen*think

15:26 gwmngilfenit's fired when the final call to /unattended/built is received, I think

15:26 gwmngilfeni'd have to check

15:27 straldi1ok I will try to use foreman_bootdisk

15:28 seanhandleygwmngilfen: that'd be good

15:28 gwmngilfenseanhandley: so if thats true, and you (a) safely run puppet in your %post / finish and (b) alter your script to sleep for 30s or so while the host reboots, you'd probably be ok

15:28 gwmngilfen*can safely

15:28 seanhandleythe rebooting isn't a problem

15:29 seanhandleywe put the host into build mode in foreman

15:29 seanhandleyat that point it's in ignore mode on nagios because the after_build hook fires

15:29 seanhandley(after_build = after build mode *starts*)

15:30 seanhandleyin Rails you can react to methods with before_ and after_ magic hooks

15:30 bettyclampCan't provision an ec2 instance: production.log just says it's rolling back and that something "failed to save". Paste is at http://pastebin.com/z2q0P2b9. Can't really figure out what's wrong. Any help would be appreciated!

15:30 seanhandleyand there's a method in managed host called :provision

15:30 seanhandleyso if you can hook onto before_

15:30 seanhandleythen you can in theory hook onto after_

15:30 gwmngilfenseanhandley: sure, but if the script to re-enable happens at the point of reboot, then there's 30s or so (or 5minutes for some stupid hardware I've seen) then you could potentially get alerts while it's still coming back up

15:30 seanhandleyunless there's more foreman code that makes that impossible

15:30 gwmngilfenhere, i mean the reboot at the end of the install

15:31 gwmngilfenobviously not the reboot into the installer

15:31 seanhandleyI didn't know there was a reboot at the end

15:31 cps0 has joined (~cps0@las-gw.ic.unicamp.br)

15:31 gwmngilfenwell you have to get out of anaconda/debian-installer somehow :)

15:32 enZer0 has left IRC (Quit: Quit...)

15:32 seanhandleywell, just from a rails perspective - we're reacting to method that get called on models

15:33 seanhandleythe managed host model has a method in it called :provision

15:33 seanhandleyso whenever that returns, it can be set to fire an after_ hook

15:33 gwmngilfenyeah, here we go, so after_build is run when the host state changes from build = false to build = true, and before_provision is the opposite, it'll fire when the build is complete

15:33 gwmngilfenapp/models/host/managed.rb:39

15:33 seanhandleyincluding provisioning?

15:33 discr33t has joined (cc130f32@gateway/web/freenode/ip.204.19.15.50)

15:34 asaleh has left IRC (Quit: Leaving)

15:34 gwmngilfenso, the last line of your kickstart/finish template will be something like wget http://foreman/unattended/built

15:34 gwmngilfenthats when it'll fire

15:34 gwmngilfenI think

15:35 jcoufal has left IRC ()

15:35 gwmngilfenbecause it's usually the last line, it;s also the point the host then reboots to get into it's freshly provisioned OS

15:35 ircuser-1 has joined (~ircuser-1@158.183-62-69.ftth.swbr.surewest.net)

15:36 discr33they guys, i'm having trouble creating a search in foreman. What i want to do is isolate the serial number fact and than search for the value. whenever i try the search (name = serial number and value = $serialnumber_value i get this error http://pastebin.com/JzA8wEuc

15:36 gwmngilfenseanhandley: so if your script can do something like backgrounding and pinging until the host comes up, then it should be ok

15:36 __endy__ has left IRC (Quit: Leaving.)

15:37 seanhandleygwmngilfen: It shouldn't need to do that

15:38 gwmngilfenwell, at the point the script fires, the host will be rebooting - how you handle that is up to you, really

15:38 gwmngilfen*before_provision script

15:38 orabin has left IRC (Ping timeout: 240 seconds)

15:39 coffmant1 has left IRC (Ping timeout: 240 seconds)

15:39 coffmant has joined (~coffmant@fw.snapbs.com)

15:39 __endy__ has joined (~Adium@212.117.65.186)

15:39 gwmngilfendiscr33t: i can't replicate that (on psql anyway) - something like name = boardserialnumber and value ~ PY works for me on the facts page

15:41 pchalupa has left IRC (Quit: Leaving)

15:43 tombar has left IRC (Remote host closed the connection)

15:43 seanhandleygwmngilfen: Box is building - I'll let you know what happens

15:43 seanhandleythanks for the help :)

15:43 gwmngilfencool :)

15:44 dustint has joined (dustin@nat/redhat/x-hpgaeysmhqledecz)

15:44 cbaal has left IRC (Quit: Page closed)

15:45 red_beard has left IRC (Ping timeout: 256 seconds)

15:45 foomatty has left IRC (Quit: Leaving)

15:46 goncalop has left IRC (Quit: Connection closed for inactivity)

15:46 bilco105hey seanhandley ;)

15:46 seanhandleybilco105: !

15:47 red_beard has joined (~thebeardi@fsf/member/thebeardisred)

15:47 mechanicalduck has left IRC (Read error: Connection reset by peer)

15:48 mechanicalduck has joined (~mechanica@37-1-162-187.ip.skylogicnet.com)

15:49 dcaplan has joined (~dcaplan@pool-98-110-153-52.bstnma.fios.verizon.net)

15:49 pophysis has left IRC (Remote host closed the connection)

15:52 Willow has joined (~Dillow@93.109.180.189)

15:52 pophysis has joined (~pophysis@ip-129-15-241-3.mba.ou.edu)

15:55 cps0 has left IRC (Quit: cps0)

15:56 suresh12 has left IRC (Remote host closed the connection)

15:56 Willow has left IRC (Ping timeout: 240 seconds)

16:00 _nick has joined (~nick@zarquon.dischord.org)

16:00 pierigHow can I use the foreman ec2 ssh key? It seems it's asking me for a passphrase

16:00 nick- has left IRC (Quit: Changing server)

16:01 coffmant has left IRC (Quit: Leaving.)

16:02 coffmant has joined (~coffmant@fw.snapbs.com)

16:02 salecss has left ()

16:03 discr33tgwmngilfen: interesting, i wonder if it's a mysql thing. what does boardserialnumber pull compared to serial number?

16:04 karimb has left IRC (Quit: Leaving)

16:06 gwmngilfendiscr33t: i have boardserialnumber = ".8PY2DT1.CN4864325N0123." for a random host

16:07 avozza has left IRC (Quit: This computer has gone to sleep)

16:07 dcaplan has left IRC (Ping timeout: 256 seconds)

16:07 mhulan is now known as mhulan|away

16:08 anthonys_ has joined (~anthonyso@197.211.219.96)

16:09 straylen has left IRC (Quit: Leaving.)

16:12 anthonysomerset has left IRC (Ping timeout: 256 seconds)

16:12 jdetiber is now known as jdetiber|lunch

16:13 discr33tgwmngilfen: os x auto-correct got the best of me once again :(

16:13 suresh12 has joined (~suresh12@122.174.101.119)

16:14 gwmngilfenaha :)

16:15 eper has joined (~eper@office-213.zen.co.uk)

16:17 taloszerg has left IRC (Remote host closed the connection)

16:20 wnkz_ has left IRC ()

16:20 anthonys_ has left IRC (Quit: My MacBook Pro has gone to sleep. ZZZzzz…)

16:21 elyezer_ has joined (~elyezer@189-12-234-86.user.veloxzone.com.br)

16:22 elyezer_ has left IRC (Remote host closed the connection)

16:22 pieriganyone know if the passphrase that is set on my ec2 ssh key is normal?

16:22 cps0 has joined (~cps0@las-gw.ic.unicamp.br)

16:23 pierigecho 'select secret from key_pairs where compute_resource_id = 6;' | psql -d foreman -t > ssh_key

16:23 pierigthat's how I pull it from DB

16:24 pierigmy issue is somewhat similar to this https://groups.google.com/forum/#!topic/foreman-users/j2p4EKXog4k but I've had SCL enabled for a long time now

16:25 sghai has left IRC (Ping timeout: 256 seconds)

16:25 pnavarro has joined (pnavarro@nat/redhat/x-gauacviocipoxkvr)

16:26 sahina has left IRC (Ping timeout: 240 seconds)

16:27 kireevco has joined (~kireevco@12.52.79.36)

16:27 itguru has joined (~gabriel@unaffiliated/itguru)

16:28 itguruI just tried installing Foreman on a brand new Amazon Linux image, and image my confusion, when it bailed out with Ruby errors. Now, I'm pretty sure that I'm not the first to hit this, and I'm wondering if any of the nice folk in here have already figured a way around this?

16:30 tombar has joined (~tombar@r190-134-31-161.dialup.adsl.anteldata.net.uy)

16:31 lamar has joined (~Adium@2001:1458:202:120::101:ce78)

16:32 lamarif I have the password of a host set directly, can I overwrite it with the password from the hostgroup?

16:32 pieriglast time I tried install on amazon linux was too painful

16:32 pierigI gave up and used a centos6.5 ami

16:33 dimatha has left IRC (Remote host closed the connection)

16:33 dimatha has joined (~dimatha@saddleless-deposition.volia.net)

16:35 coffmant has left IRC (Quit: Leaving.)

16:36 cps0 has left IRC (Quit: cps0)

16:36 walden is now known as walden|brb

16:37 cps0 has joined (~cps0@las-gw.ic.unicamp.br)

16:37 mbacovsk__ has left IRC (Ping timeout: 260 seconds)

16:38 dimatha has left IRC (Ping timeout: 240 seconds)

16:38 flakrat has joined (~flakrat@pdpc/supporter/active/flakrat)

16:38 judd7 has joined (~quassel@184.20.247.12)

16:40 Guest67569 has left IRC (Ping timeout: 250 seconds)

16:40 kbidarka has left IRC (Remote host closed the connection)

16:42 raul has joined (~raul@62.0.168.213.sta.estpak.ee)

16:42 raul is now known as Guest31735

16:43 itgurupierig: I'm not far away from the same conclusion

16:44 ayoung_ZZzzZZzzz is now known as ayoung

16:44 momomomomo has joined (~momomomom@c-107-2-92-157.hsd1.mn.comcast.net)

16:47 taloszerg has joined (~taloszerg@c62e6602.cst.lightpath.net)

16:48 lamar has left IRC (Quit: Leaving.)

16:51 witlessb has left IRC (Ping timeout: 256 seconds)

16:52 walden|brb is now known as walden

16:52 Willow has joined (~Dillow@93.109.180.189)

16:54 gwmngilfenitguru: it's pretty much impossible to help fix errors without being shown what the errors are. That said, Amazon Linux is not an officially supported distro for the installer, although we're happy to take patches in general

16:56 taloszerg has left IRC (Ping timeout: 260 seconds)

16:58 Willow has left IRC (Ping timeout: 272 seconds)

16:58 DaveIW has left ()

17:02 witlessb has joined (~wb@cpc29-lewi15-2-0-cust72.2-4.cable.virginm.net)

17:04 seanhandleygwmngilfen: So, the before_provision fires before the reboot

17:05 RobertBirnie has joined (~RobertBir@157.22.52.2)

17:05 __endy__ has left IRC (Quit: Leaving.)

17:05 seanhandleyand having the script wait for the host to come back up is no good because once it's up puppet starts running

17:05 seanhandleyand we need puppet to have finished running

17:06 Claer has left IRC (Quit: Leaving)

17:06 gwmngilfenthen you're into plugin territory - foreman_hooks involvement ceases once we get to that reboot

17:06 larryone has left IRC (Quit: This computer has gone to sleep)

17:07 gwmngilfeni have a plugin which takes action upon receiving an agent's ENC request (in my case, it's for enfocing a default hostgroup when hosts don't have one set)

17:07 gwmngilfenyou could hack on that to trigger something

17:07 gwmngilfenor just script the api calls to to the hosts reports until you have >0 reports

17:07 sthirugn is now known as sthirugn-lunch

17:08 seanhandleytbh if foreman gets a report for a host then we can trigger the API call to tell nagios to pay attention

17:08 eper has left IRC (Ping timeout: 264 seconds)

17:08 seanhandleyit doesn't matter if that call happens more than once

17:08 seanhandleycan you use foreman hooks to react to report creation?

17:08 gwmngilfeneven so, thats defintely not a hook that foreman_hooks currently has, so you're looking at some form of extension

17:09 gwmngilfenno, it doesn't i don't think

17:09 seanhandleyhmm

17:09 seanhandleyI've shaved enough yaks this week already

17:12 suresh12_ has joined (~suresh12@122.174.78.88)

17:12 gwmngilfenif you need something *now* then my suggestion would be to laucnh something from the before_provision hook which backgrounds, loops over checking the reports API, and then updates nagios when either (a) a report comes in) or (b) a timeout is hit

17:13 larryone has joined (~larryone@178.167.254.21.threembb.ie)

17:13 gwmngilfenif you can wait, i'd suggest a feature request (or better, a patch :P) to enable what you need

17:13 gwmngilfenthe author of foreman_hooks is away all week, just fyi

17:13 lamar has joined (~Adium@AAnnecy-158-1-131-32.w86-200.abo.wanadoo.fr)

17:14 pophysis has left IRC (Remote host closed the connection)

17:14 suresh12 has left IRC (Ping timeout: 264 seconds)

17:16 tmclaugh[work] has joined (~tmclaugh@64.119.157.115)

17:16 coffmant has joined (~coffmant@fw.snapbs.com)

17:19 larryone has left IRC (Ping timeout: 240 seconds)

17:20 seanhandleythanks gwmngilfen

17:20 seanhandleyYou've been massively helpful :)

17:20 seanhandleyI'll mull over the options with the dev team

17:21 acidrainfall has joined (~adavis@ip98-176-100-253.sd.sd.cox.net)

17:21 acidrainfallHey guys

17:21 acidrainfallQuestion

17:22 acidrainfallI have two sets of modules - one is in /environments/(environment)/modules, the other is in /modules off of confdir

17:22 acidrainfallForeman sees the environment-specific modules and their classes, but not /modules