David Sprayberry, NOIRLab
August 25, 2022
After the Contreras Fire swept over part of Kitt Peak in June, July was devoted to restoring electrical power and water service to the mountaintop. We also carefully assessed every structure for safety, possible fire damage, and the extent of cleaning required. Currently electricity at the summit is provided solely by back-up generators (more on that later). Because the water system lost pressure during the power outage, it was necessary to flush it with chlorine and then fresh water from the storage tanks, and then have the water tested to ensure it meets EPA standards for safe drinking water (it passed!). Work at the telescopes during July was limited to skeleton crews going up for specific tasks to prevent further issues with sensitive scientific equipment. At the Mayall, this included restarting the building’s cooling system, changing filters in the purified dry air system that purges the focal plane enclosure, and changing filters in the HVAC system serving the spectrograph clean room (fondly known as “the Shack”). We also performed regular inspections in all the NOIRLab domes for leaks and electrical faults.
Beginning August 1, telescope staff were allowed to resume almost normal on-site work. At the Mayall we spent the first week cleaning everything we could find a way to reach inside the dome, and any other areas potentially affected by ash or smoke intrusion. The interior of the dome wasn’t particularly dirty, but we cleaned everything thoroughly anyway. We also removed the protective tarp and plastic sheeting from the primary mirror and prime focus corrector, and made a preliminary inspection of the optics, which look fine. During the second week of August we caught up on overdue preventive maintenance of the telescope and dome, and worked to provide a low-bandwidth internet connection to the Mayall.
Road Conditions
Throughout August, road and weather conditions have significantly limited the time our team can spend on-site. The fire damaged guardrails along the road, and there has been a lot of erosion from burned areas washing across the road during heavy rains. This extraordinary erosion has clogged many of the drainages that cross the road, sometimes causing impassible mudflows. Additionally, the fire damaged several drainage culverts under the road. The Arizona Department of Transportation (ADOT) is making progress on repairs, but they are hampered by frequent heavy summer “monsoon” thunderstorms since late July. Finally, erosion is worsening the normal amount of rockfall along the road during storms. Combined with this year’s strong, frequent storms, we are encountering more rocks and boulders in the road.
Road restrictions are still in place. Repair work frequently requires closing both lanes, and so that we interfere as little as possible ADOT requires us to go up and down the mountain in one convoy at the same times every morning and afternoon, weather permitting. NOIRLab is also concerned for site safety due to the possibility of road closures caused by mudflows or rockfall. Whenever rain begins or is threatening, everyone on the site ceases work, makes their work site safe, and assembles for departure in an early convoy.
Electrical Power and Internet
The fire damaged between 10 and 20 utility poles that bring electricity and fiber optic data to the summit. Our utility provider, the Tohono O’odam Utility Authority (TOUA), is working hard to replace the damaged poles, but they face a number of obstacles. Many of the poles are in very remote locations. Other lines and poles elsewhere on Tribal lands were also damaged by storms. And like everyone else TOUA is grappling with supply chain disruptions. It will be at least several weeks until power is restored, and then fiber optic lines will be replaced.
The observatory is currently powered by on-site backup generators. One serves the Mayall and adjacent University of Arizona buildings, and another serves the rest of the mountaintop. This is expensive in terms of refueling needs, and leaves the site without back-up power sources if one of the generators were to fail. We are in the process of renting additional generators and connecting them to use as primary power sources. Once this is done the permanent back-up generators can return to back-up status.
Keeping the generators fueled is a challenge. Not all vendors will deliver diesel fuel to our remote site. Recently our normal supplier was unable to make our regular delivery when their truck broke down, and the Mayall generator ran out of fuel before an alternate supplier could reach the summit. Storms and road conditions further complicate the situation.
For temporary internet service DESI has obtained a Starlink system and service for the Mayall. The equipment is installed and connected to one network switch in the Mayall building, but that connection requires a standalone DNS server in the Mayall to allow normal access to the many other computers and switches in the building. DNS within NOIRLab is normally provided by one central server in Tucson, but that is obviously not an option right now, so work is underway to construct this standalone DNS system.
Optics Cleaning
We want to clean both the primary mirror and front surface of the prime focus corrector with carbon dioxide “snow”, and do a wet wash of the primary mirror. However, these operations require a relative humidity of < 50%, sustained through the day, to prevent condensation after the “snow” cleanings and promote drying after the wet wash. We haven’t seen a relative humidity reading that low since the August 1 return to the summit. DESI very much wants the optics cleaned before going back on-sky, as relative throughput measurements using the guide cameras are the first tests called for. We’re waiting for the first relatively dry day to seize an opportunity for optics cleaning.
Restarting DESI
Restart of the DESI equipment can’t really begin until some internet connectivity is restored and remote monitoring of the equipment’s health is possible. The expected order of events is to re-establish a fully-functioning network with internet connectivity; then bring the DESI computer cluster back on line; and then change filters inside the focal plane cooling system, restart focal plane cooling, and test basic focal plane and fiber view camera functions.
Once the focal plane guiders are running we can perform a simple throughput test if the optics have been cleaned and observing conditions are good. The next major phase will be restoring spectrograph operations. This involves restarting the cryostat control systems, pumping down cryostats, turning on the cryocoolers, and waiting for the CCDs to reach working temperature. The spectrograph restarts do not depend on weather conditions, but we do need a back-up electrical power supply or to know that one is on the way, so we don’t risk another unplanned warm-up were the back-up generator to fail.
We don’t yet have a firm timeline for restarting DESI, as so much depends on the weather and other things we have little or no ability to control. We’re doing the best we can, and will continue to post updates as developments warrant.
