Watchdog for asterisk: implementation (Part VII)

Part VII: Watching the Watchdog

It's good to know if a circuit is functioning properly, so having the LED indicators was very helpful. The LEDs are also nice for showcasing the gadget to other people. Now the watchdog box is located in my garage, and I have been checking on it daily, morning and evening, whenever I am home and not doing much. Who knew that I would become the watchdog for the watchdog!!

The other day I found an old webcam that I bought at a clearance sale. It's a Creative Labs Vista webcam. Though having the name Vista, it predated Windows Vista and in fact is not compatible with it. I had used it for a short while with my XP laptop, but it's been just sitting in the drawer doing nothing. So I thought of (geeky, to be sure) using it to save me some efforts in checking on the watchdog box.

I found a linux driver for the webcam and installed it on the same linux machine that hosts my asterisk telephone server, and there's also a simple webcam GUI program from the same website. The installation was easy, and in just a few minutes I was looking at the watchdog box via a VNC window to the linux machine! Here is a screen cap of the watchdog when it's not detecting a tone:


Here is another screen cap, when a tone is detected:


I suppose my next project will be a video analytics program to monitor the watchdog and send some kind of alert when the watchdog appears not to be functioning ...

Watchdog for asterisk: implementation (Part VI)

Part VI: Coping with Changing Temperatures

Ceramic capacitors come with nominal values, but there is typically a relatively large range of tolerance (e.g., 20%). In addition, its effective capacitance varies with temperature. This is a particularly nasty problem for timing circuits whose parameters are based on RC. This plainly titled article explores some of the mechanisms of capacitance variations. Here is a short white paper with information on the temperature coefficients of ceramic capacitors, including an explanation on the various designations (X5R, Y5V, NPO, etc.). Ah, things one never quite learns in school ...

The two subcircuits of the watchdog, namely, the tone detector and the missing-pulse detector, each have an RC timing network. For the tone detector, the RC value determines the center frequency of the tone. For the missing-pulse detector, the RC value sets the time-out period when no pulses are seen at the input.

Since the time-out period is not required to be accurate (45 seconds or 55 seconds doesn't really matter, since we'll be sending pulses in 5-10 second intervals), we can use a fixed resistor (1Meg) and a fixed capacitor (47uF). For the tone-detector, on the other hand, a higher accuracy is required, even though there is a bandwidth tolerance. A trimpot was used to allow some adjustment during prototyping, and its setting is left alone after that.

As I anticipated some variation in the center frequency due to changing temperatures (on a typical day, the high and low temperatures of location of my linux server can easily have a range of 20-degrees centigrade), I generated sine-wave tones of 0.1 seconds each, ranging from 18950Hz to 20000Hz, in 150Hz intervals. The reason that these frequencies are higher than what I had during prototyping and tuning is that the location of the finished product is hotter, thereby decreasing the effective capacitance of the capacitance that sets the center frequency. In addition, the temperature becomes higher during the day; so at least one or two of these tones should be detectable by the tone decoder.

Here is the asterisk_watchdog script:

#!/bin/sh

TONES_DIR=/etc/asterisk/watchdog/tones
INTERVAL=5
amixer set 'Master' '100%'
amixer -- set 'PCM' '-4.50dB'

run_watchdog()
{
while (true)
do
  if [ -f /var/run/asterisk.pid ]; then
    ASTERISK_PID=$(cat /var/run/asterisk.pid)
    RUNNING=$(ps auxw|egrep '^[^ ]+ +'$ASTERISK_PID' '|grep -c asterisk)
    if [ $RUNNING = "1" ]; then
        for tone in ${TONES_DIR}/*.raw
        do
          aplay -q -f cd $tone
          sleep 0.5
        done
    fi
  fi
  sleep $INTERVAL
done
}

run_watchdog &

The amixer settings were derived by trial and error. At these settings with my raw files (magnitude 0.45 from audacity), the tone decoder seems to work the best.

The "*.raw" files are raw (headerless) samples that can be played by aplay with -f cd. Initially, I generated one .raw file for each frequency, and by making the watchdog script read from the directory of raw files, I can easily add and remove additional frequencies. However, when I was satisfied with the set of frequencies, I combined these all into one raw file, with 0.35 second of silence in between. In the end I just need one file, so that aplay won't be called multiple times per iteration.

Note that the main loop is encapsulated in a shell function, and the last statement of the script is an invocation of the function followed by the backgrounding ampersand &. This is a nifty way of running a shell script as a daemon. I also wrote (copied-pasted-modified) an asterisk_watchdog init.d script and placed it under /etc/init.d, and added it to the list of daemons to be started at regular runlevels (2, 3, 4, and 5) via chkconfig.

And now the watchdog is running. It's been running pretty reliably for more than a week now, and I am very happy with the result.

Of course, one question remains: who will watch the watchdog? (See next part.)

UPDATE (2009-07-15): I stumbled upon this neat article from EDN that uses a "digital variable resistor" (actually a more complex device than the name implies) to perform temperature compensation for a voltage regulator. It would be an overkill for this project to use this device (I might have chosen a better capacitor instead), since I can achieve my objective through software. Nevertheless, I thought the design in the article is pretty cool.

Watchdog for asterisk: implementation (Part V)

Part V: Putting everything together

Like most projects, this one was both late and over budget. Having not soldered any electronics for a long, long time, I messed up quite badly on my first attempt. Here is the incomplete and abandoned board. I won't even show the other side of the circuit board.


Not only did I mess up the board, I also did some damage to the tip of the soldering iron. The tip was badly oxidized and eaten away. Later I found out that the combination of a high-wattage (60W) iron with a "no-clean", lead-free solder was not particularly good. This document has some informative notes on caring for the tips, and I should've read it before venturing into my own way of doing things.

Though the first attempt was abandoned, I realized that the circuit needed some modification (see Part IV). And since I am still not very skilled at soldering, perhaps it's better that I minimize the amount of soldering that I have to do. The single-pad-per-hole boards are just too troublesome to deal with. As I looked for a board that would be easier to use, I found this at Radio Shack:

This board is 1.875" by 2.875", just the right size for the number of components of my circuit (now with a regulator and a couple more capacitors). It also fits quite nicely into an electrical box, which will be covered with a two-jack faceplate (see below). I also laid out the components in such a way that the LEDs can be seen, and the connectors for the relay and pads on the telephone be fastened, by drilling holes on the box. In addition, a set of PC board standoffs proved to be useful for mounting the board inside the box.

Here is the final circuit board. The two blue connectors on the lower right corner are for the relay and for two wires that form a switch that is closed when the handset of the phone is on-hook and open when it's off-hook. This is so that the phone can be used once it's off-hook even if the watchdog timer is still active. I failed to fit the two connectors together as designed, so they were mounted on the board somewhat crookedly. Next to the connectors are the two LEDs: the green one is on the output of the 555 timer; when everything is in order, it should be on all the time. The orange LED is on the output of the 567 tone detector (also serving as its pull-up). It will be flashing at an interval programmed into the asterisk watchdog process on the Linux machine. On the left hand side is an LM7805 5V linear regulator, providing the Vcc for the two ICs.


Here is what the board looks like when it's mounted in the box:


Note that the relay is mounted on the other side of the box, with wires for the coil connected to the terminal connectors on the right. The other two wires, soldered to the normally closed terminal and the common terminal of the relay output, will be connected to the two-jack faceplate, so that when the relay switches on/off, the two jacks of the faceplate will be connected or disconnected. I needed to make sure that the polarity of the connection between the two jacks is reversed: i.e., red to green and green to red. This is because telephone line cords switch the polarity (kind of silly), and when one uses two telephone cords together in this fashion, there should be an additional polarity switch so that the total number of reversals is odd. On the top side of the box, there is a row of holes that I drilled for displaying the LEDs and for tighting the connectors' screws. I made a mistake by drilling the holes in the wrong orientation, so I ended up with extra holes. It's OK, however, as the extra holes can serve as vents.

Finally, here is the watchdog circuit in service:


To see the green LED light up, I took this picture in the dark:


I will discuss the shell script, asterisk_watchdog, in the next part.

UPDATE (2009-07-17): Getting the right polarity on phone lines can be tricky. The phone cords that come with telephones typically are of the crossover type, i.e., the polarity of the pins are reversed from one end to the other. However, there are also some cords that look and feel exactly line regular phone cords, but they are not crossover.

I didn't realize this until I used a phone line tester and tested the various jacks that I now have: the watchdog output and the two wall jacks that are driven by the asterisk FXO interface. (The phone line tester was surprisingly hard to find, not even at the local Radio Shack. I stumbled upon one at Ace Hardware; it was overpriced at $10.49, but I bought it anyway since I was in the mood of getting all my wiring right.) Much to my chagrin, the tester showed red (polarity reversed) on the watchdog output jack and one of the two wall jacks. It showed green on the other wall jack. The primary input, i.e., the jack that's directly connected to the network interface, also showed green -- that was a relief.

Interestingly, the FXO interface on the asterisk card also gave a red reading. After examining the cords carefully, I realized that I was using a patch cord for the watchdog input, a crossover cord for the wall jack that showed red, and a patch cord for the wall jack that showed green. So the solution was simple: swap the cords for the watchdog input and the first wall jack. Voila, now all jacks for the terminal devices are green!

Watchdog for asterisk: implementation (Part IV)

Part IV: Revising the circuit to use a transistor to drive the relay

Well, it turns out that I needed to revise the circuit since I'll be using an unregulated power supply---a simple class 2 transformer (like the ones used for charging cell phones or powering computer peripherals). I have one with adjustable voltage (3V, 4.5V, 6V, 7.5V, 9V, and 12V). At nominal 4.5V, the measured open-circuit output is actually about 7.5V, and that is sufficient for my relay (Radio Shack Catalog #275-005) , which is rated at 7V to 9V.

But there were a couple of problems with unregulated supply. Since the voltage will vary with the amount of load, it could affect the accuracy and reliability of the operations of the two ICs on the circuit. Since the LM567 is rated at 9V maximum, I decided to use an LM7805 5V linear regulator. But this gives rise to another problem. Although the output of the 555 timer can drive 200mA, its output voltage is about 1.3V below its Vcc, in this case 5V. Secondly, driving an inductive load (relay) is tricky. The circuit in Part III is actually not quite right, because even with the protective diode, the negative voltage of -0.6V (the kickback voltage of the relay clamped by the diode) on the output of the 555 can actually cause it to malfunction. The application note for the NE555 (Philips AN170) (p.7, Figure 8) recommends that a series diode be connected between its output and the inductive load, but that means another drop of 0.6V, which is far from sufficient for driving the the relay.

The solution is to use an NPN transistor (Q2), with the output of the 555 driving the base (via a resistor [R6] to get the necessary base current). The current to drive the relay will be supplied from the unregulated output of the transformer. The modified circuit is shown below. Note that since Vcc is now 5V, I reduced the value of R5 so that D2 will not be too dim.

Watchdog for asterisk: implementation (Part III)

Part III: Watchdog Timer (Missing Pulse Detector)

The following circuit implements a watchdog timer, sometimes also called a missing pulse detector.


The circuit expects pulses (active low) at intervals less than its timing constant, given approximately by (see LM555 datasheet):

T = 1.1 * R4 * C8

When a pulse is present at the input to this circuit (which is the output from the tone decoder), capacitor C8 is discharged because Q1 is turned on. It will again be charged starting from 0V, while the output (pin OUT of the 555 timer) remains high. Thus, it takes time T for the capacitor to be charged to the triggering threshold voltage for the output to go low. If the next pulse comes long before this amount of time elapses, then we start the timing cycle again without changing the output.

For the values chosen (1MegOhm and 47uF), T is about 52 seconds. Thus, the host computer needs to send a tone blip to the decoder no more than every 50 seconds or so. I plan to make that 10 seconds, with pulses of slightly varying frequencies (from the low-cutoff frequency to the high-cutoff frequency of the tone decoder), so that we can accommodate some shifts in the center frequency due to temperature variations.

D3 is a flyback diode that absorbs the voltage surge when the relay is turned off; this is necessary to protect the 555 IC. The output of the relay is used to connect or disconnect a plain telephone from the phone line. SW1 is connected to the handle of the telephone, so that when it is picked up, the relay will lose its current in that event as well. Note that the relay is connected as normally closed, so that when we lose power to the relay by whatever reason (power outage, host computer crash, or handset picked up), we get the plain telephone online.

The video below demonstrates the watchdog timer in action. For the purpose of this demonstration, I have replaced R4 with a 100kOhm resistor, so that the time constant T is now about 5 seconds. Eight tone blips of 0.25s in width in 2s intervals are generated from the host computer. The tone decoder nicely lights the orange LED accordingly, and the timer turns on the green LED and make it stay on as long as the blips come in frequently enough. About 5 seconds after the last pulse, the timer turns off the green LED and, in the final circuit, the relay as well, putting the telephone online.

Watchdog for asterisk: implementation (Part II)

Part II: Calibrating the Tone Decoder for Specified Frequency

The tone decoder schematic is given below (Vout is connected to the input of the timing circuit, to be described in Part III):


(Thanks to Linear Tech for providing a nice free simulator LTSpice, which I used to produce the above drawing. The IC is actually an LM567 equivalent, though the schematic has Linear Tech's logo on it.)

I have constructed this circuit on a breadboard for testing. The 10k potentiometer (R1) on the test board is from the kit that I purchased from NightFire. It has only one turn, so it is probably not suitable for production as it might be less precise. I have also bought a couple of multiturn trimpots from Fry's Electronics, which I will use in the final circuit. The following video show the calibration of the tone decoder. (The right half of the board is the timing circuit, which we will ignore for now.) As I turned the trimpot slowly while the test tone is being played from the host computer, the orange LED eventually lights up when the frequency is matched.



Related posts:
Part I
Part III

Watchdog for asterisk: implementation (Part I)

Part I: Generating a tone from host computer for an "alive signal" for the watchdog

First, it is obvious that a mini-stereo plug is needed to get the signal out of the sound card. I managed to find an old headset from my junk pile, and carefully cut out the mini-stereo plug and the cord for the earphone part. Stripping the wires is a little tricky, as the inner wire is very thin.

Next, I need to generate a tone. I picked 18kHz, which is almost inaudible, though the LM567 part can easily handle a wide range of frequencies. I also need to adjust the amplitude of the tone so that it is sufficient to be detected by the LM567. The datasheet for LM567 says that the minimum detectable input voltage is about 20-25 mVrms, but in one of its example circuits (a touch-tone decoder) an input voltage level of 100-200 mVrms is indicated. So a minimum of 100mVrms should be safe.

I used Audacity, which provides a nifty tone generator, to produce 30 seconds of 18kHz sound:


The next dialog box allows one to specify the frequency, amplitude, and duration:


Since I don't have an oscilloscope, I had to rely on my very primitive multimeter, which can measure AC voltage to only an precision of 0.1V. After experimenting with a few values of amplitude and measuring the corresponding voltage, I decided on an amplitude of 0.4. During this measurement, the volume setting in the Windows (Vista) mixer is set to its full value 100. This is because, when the circuit is deployed, the signal on the sound card in my Linux box will be produced via an ALSA program directly reading from a PCM WAV file, without final gain adjustment. Of course, I may need to readjust the amplitude since the output signal level on the Linux box may be different from my Windows box.

Watchdog for asterisk

I've been mulling over designing a watchdog circuit for my asterisk (open-source telephony) process that runs on my linux server. My current safety mechanism consists of a relay that connects a vanilla telephone directly to the POTS line in the event of a power failure. However, if the operating system crashes, there is no way to switch the relay automatically as in the case of power failure.

There are several ports out of my old linux box that I can use to communicate to the outside world: a RS-232 serial port and a couple of USB ports. The RS-232 serial port is taken by the UPS, so it's out of consideration. USB is too complex and probably requires some expensive parts just to get started (cf. this article on linuxfocus.org).

There is another possibility: the sound card. Since the server doesn't care for outputting music or other sounds, I can program a watchdog process to monitor asterisk and output a tone, say every minute. If the operating system crashes, there will be no such notification tones. A timer circuit can then turn on the relay to enable the vanilla telephone.

I'll need a LM567 (or equivalent) tone decoder and a 555 timer for this purpose. A simple 567 circuit is found here. The timing function can be implemented by this circuit using a 555. To generate the tones through the sound card, this is probably the way to go.

Will update when I actually construct the circuit and make it work.

UPDATE 2009-06-01: Don't forget a flyback diode (1N4148?) across the relay!

UPDATE 2009-06-05: NightFire Electronic Kits looks like a good place to get the parts I need.