hacker emblem
jaegerfesting
Search | Tags | Photos | Flights | Gas Mileage | Log in

Public Safety

Started: 2020-10-20 19:48:52

Submitted: 2020-10-21 00:33:47

Visibility: World-readable

In which the intrepid narrator enjoys a Public Safety Power Shutoff triggered by potential fire weather

Two years ago, local investor-owned utility PG&E's transmission lines sparked the Camp Fire, which exploded into a massive conflagration and destroyed the town of Paradise around the time I flew to Sacramento for the Logan family reunion. California courts found PG&E legally responsible for the entire disaster, despite the fact that the blaze wouldn't have exploded as it did absent a century of aggressive fire suppression and forest mismanagement compounded by anthropogenic climate change, overwhelming emergency responses and leaving dozens dead.

Faced with massive liability for any new fires sparked by under-maintained power lines in a new hostile legal climate, PG&E did the least-bad thing it could think of: turn off power for people in rural areas, or people whose power is transmitted through rural transmission lines, whenever weather conditions suggested high fire danger. PG&E called these "Public Safety Power Shutoffs".

This name struck me as especially Orwellian, but it took me until this week to figure out why: A literal interpretation of "public safety" in Chinese is 公安, which is what the People's Republic calls its police forces, which are at least as interested in suppressing internal dissent as they are in other crime.

Power lines grounded for tree trimming
Power lines grounded for tree trimming

My power was out for three days in August, an unscheduled outage triggered by a series of lighting storms that rolled in from the ocean. My power went out again for an afternoon in September when PG&E cut power to our line (and grounded it just to be sure) to let a team of arborists cut down trees encroaching on the power line in between the switch-backs on my driveway. They left behind a field of stumps on the hillside under the power line and a crude trail cutting through the brush, descending the fall line down the steep hillside.

Stumps under power line
Stumps under power line

I finally got to experience my own Public Safety Power Shutoff in the middle of October. PG&E telegraphed the planned outage in advance, regretfully informing us that it was sadly necessary to cut off our power in order to keep them from being liable for setting off any fires. This outage, we were assured, was limited in scope to only the small number of people who lived at the very top of the ridges, where the wind would be the strongest and the transmission lines were in the greatest danger. This, unfortunately, included my house. (Being warned in advance did give us a chance to eat the ice cream in the freezer in advance, so it wouldn't regretfully melt when the power went out.)

Since our last power outage I bought a UPS for my network. I have a line-of-sight microwave connection, with an antenna on my roof pointing in the general direction of Scotts Valley. (Its status page, accessible on my local network, tells me that the air delay is 89525 ns, or "approximately 8.333 miles".) I measured that I need 21 watts to power the radio and my router, or 33 watts total including the tiny Linux machine I use as my network storage, backup hard drive, and utility server. I bought a UPS advertising 1000 VA; its spec sheet suggested I could run my network for about two hours on battery.

(Along the way it occurred to me to wonder if I could get more efficiency out of my UPS if I could get a DC output to power my devices directly, because they mostly accepted 12 volts at various power. This does seem to be a real thing — though since it targets commercial network infrastructure, maybe not at my price range — and maybe my next UPS will be a DC UPS.)

I got the chance for a live-fire test my UPS on Tuesday evening, the day before the Public Safety Power Shutoff was scheduled to begin, when the power went out for a couple of minutes. This was long enough to confirm that the UPS was indeed able to power my network, though I was somewhat disappointed by the granularity of the information I was getting out of the UPS' management port; and I couldn't help but want to set up a monitoring infrastructure to replicate Monarch (the monitoring infrastructure I used inside Google) and Viceroy (the graphing and reporting system) so I could get detailed time-series graphs of line voltage and battery status and power consumption and time remaining.

PG&E planned to shut off my power between 20:00 and 22:00 on Wednesday, 14th October. The power went off, as promised, a few minutes after 20:00, while I was doing some final checks on the UPS. I switched immediately to my head-lamp, and headed downstairs to monitor the effect of the outage on the UPS.

Dining table lit by candlelight
Dining table lit by candlelight

As I expected, the UPS was able to run my network, providing wifi and a stable connection to the Internet. I monitored the state of its battery, then eventually powered down my utility server to reduce the load on my battery. (I've resisted switching too much of my network services over to the utility server, because of cases like this where I can shut down the server and get more time on the battery for the rest of the network, but maybe that's optimizing for the wrong things.)

The UPS held out for about two hours before running out of power, with a spectacular alarm that woke up Calvin because the network infrastructure is in his room (because that's where the other end of the cable to the microwave antenna went, for reasons I don't totally understand but probably involved having an office up there; it does give my wifi router a commanding view of the house). I turned off the UPS, then went to bed early, because there wasn't much else I could do in the dark.

Thursday morning our landlord-neighbors turned on the diesel generator hooked into the electrical panel between our houses. This gave us enough power to run most of our lights and fans and computers, but we couldn't run the AC, and that proved problematic later in the afternoon as the temperature in my office-bedroom (the warmest part of the house, because it gets direct afternoon sunlight and the air circulation with the rest of the house is poor at best). I could hear the generator droning on all day in the background until I put on noise-canceling headphones (which also blocked the soft whine of the fan at my feet, circulating the air in the room in an ultimately-futile attempt at keeping me cool enough to work).

The kids' school was included in the planned outage, and the school sent an email to say that they would be running on generator power and if anyone wanted they could drive to the parking lot and use the wifi to attend class (remotely) or do homework (asynchronously). But by Thursday evening they followed up with another email that the generator was in danger of overheating so if the power was still off on Friday (as originally scheduled) they'd cancel synchronous education on Friday and assign homework to be done asynchronously. (Anyone who didn't have power at home could get an excused absence and make up the homework later.)

PG&E brought in an army of line crews to inspect the transmission lines and decided they could restore power for most customers, except for the furthest-outlying areas at the very top of the ridges. This, of course, included me (but at least the school had power, so they could maintain their schedule of remote classes).

The generator went off overnight (so I didn't have to hear it droning on while I was trying to sleep), and the UPS gave another hour of Internet access in the dark before I turned it off and went to bed.

By Friday morning the status page suggested we were going to get our power back sometime in the morning. (I signed up for PG&E alerts for my entire zip code, but I never actually received any alerts during the outage, or when the power was restored.) I saw at least one helicopter flying low over the ground, apparently following the precise line of the power lines, possibly performing a visual inspection of the lines before deciding it was safe to restore power. The power came back before 09:00, and we switched back to grid power with only the slightest interruption in power, bridged by the UPS on the network and the batteries on our computers (with the exception of my new iMac; I solved that potential problem by not turning it on until we switched back to grid power).

I have now survived my first Public Safety Power Shutoff, one of the many hazards of living in the mountains, and I am at least relieved that our UPS and generator helped to get us through the outage.

threads are trick. Yessss...tricksey threadesess.
They runsses when we don'ts expects them, yesss.
Nasty, tricksey threadses.
- Zan_Lynx, Mass IRC, 13 January 2003