aberti

Recovering from the Contreras Fire

Benjamin Weaver, NOIRLab

January 24, 2023

DESI collaborators have previously reported on the Contreras Fire and how it initially affected DESI operations along with an update as personnel were able to return to Kitt Peak National Observatory (KPNO). Even as we were cleaning up KPNO and the DESI instrument in August 2022, we knew that it would possibly be several months before the Tohono O’odham Utility Authority (TOUA) could replace the infrastructure bringing electrical power and line internet (a fiber-optic link) to KPNO. In fact, line internet was not restored until December 7, 2022, almost exactly six months after the start of the fire.

The story of restoring electrical power and the generators used while restoration was in progress is definitely worth telling, and far more lengthy and detailed than the story I’d like to tell now, recovering from the missing internet connection.

The DESI Data Management team, working closely with the on-site NOIRLab personnel, developed a plan to work around the missing internet, even as DESI returned to full-night operations. It wasn’t a new plan; in fact, it was a very old plan: “Sneakernet”. The idea is that a human being (presumably wearing sneakers) can hand-carry data on physical media from one location to another, and in certain circumstances, the effective bandwidth of that transfer can be much, much higher than the bandwidth of an internet connection or other communication channel. This is actually even more true now than it was in the past, because the storage density of physical media has experienced exponential growth similar to that of Moore’s law for CPU performance. However, while Moore’s law has been showing signs of reaching fundamental physical limits, storage density trends continue to rise.

With that in mind, we purchased six 2 TB SanDisk solid-state drives (SSDs), and a few fast USB cables. There is nothing exotic about the hardware; these items would be available in many consumer electronics or department stores. Then with the 6 disks initially carried to KPNO, we would place an entire night of data (or sometimes several nights) on a SSD. One of the personnel working at KPNO would be designated to carry the SSD to Tucson at the end of the work day. Then we would pick up the disk either in person or from a drop box and copy the data to NERSC where it could be processed with the usual DESI data pipeline. Since we purchased six SSDs, that meant we always had several SSDs on standby at KPNO in case an already-transferred SSD could not be brought back from Tucson to KPNO the next day. On weekends, no one would be available to carry the SSDs, so we simply had several nights on Monday evening’s SSD.  From the point of view of the pipeline, the only difference was that an entire night (or several nights) would arrive all at once, after a delay of 1 to 3 days, instead of the normal operations procedure of one exposure at a time, almost immediately after the exposure is completed.

The DESI instrument resumed taking data starting with night 20220825 (August 25, 2022). The first SSD was brought to Tucson on September 2, 2022. This process continued until line internet was restored on December 7, 2022. The original data transfer system was ready to go at that point and was reactivated almost immediately after line internet was restored. Thus the last night of data transferred by sneakernet was 20221206 (December 6, 2022). In total, 16,726 exposures over 100(!) nights were hand-carried to Tucson. This amounts to 6.1 TB of data.

The bandwidth of hand-carried SSDs is impressive, but so is the bandwidth of modern USB cables. After some experimentation, we discovered that we could transfer data directly from the external SSD to NERSC via Globus Online, with bandwidth performance indistinguishable from the case where the data were first copied to a workstation, then transferred via Globus. In other words, the limiting bandwidth was the internet itself between Tucson and NERSC, not the internal systems of the receiving workstation.

While we were transferring DESI data with a very old system (but with the latest SSDs), we were also experimenting with a new one: Starlink. In fact there were at least two separate Starlink installations active at KPNO during the recovery.  One, which we’ll call the “DESI Starlink”, was a strictly limited command and control channel for use by instrument experts. The DESI Starlink was not used for any significant data transfer. The other, which we’ll call the “NOIRLab Starlink” was used for more general access to all of KPNO. We did use the NOIRLab Starlink for some data transfer, with mixed results.

At least currently, Starlink works best with a point-to-point style networking connection, optimized for individuals or residences, much like a residential cable modem. Since the uplink bandwidth is currently a factor of 10-20 times slower than the downlink bandwidth, it is less practically suited as a two-way connection between two substantially-sized networks, such as between KPNO and the Tucson base facilities. This manifested as a significant, recurring routing problem. Although bandwidth was adequate for interactive use and for the transfer of small data files, the connection was frequently not working and the routing systems had to be monitored constantly. Nevertheless, we were able to use the NOIRLab Starlink to maintain an off-site copy of the DESI operations database. This copy would have been impractical to maintain using daily SSD transfers, because the copy is a real-time mirror of a live database; it is more of a stream than a set of files. There were occasions where the database mirror fell behind due to interruptions in the Starlink connection. In these cases files created by the database were transferred manually on the SSDs and then once in Tucson the data files were manually added to the database mirror.

It was interesting and a good experience to transfer data over sneakernet. It was also a lot of work and hours. I want to give credit to a big team who all contributed to this effort: Keith Blaine, Paul Demmer, Matthew Evatt, Steve Grandi, Bob Marshall, Rod Rutland, David Sprayberry, Christopher Stone, Bob Stupak (NOIRLab), Martin Landriau (LBNL), and Klaus Honscheid (OSU). Here’s one job we hope we never have to do again!

January 18, 2023

The all-sky camera on the Mayall telescope were the DESI instrument is located provides a view of the entire sky above Kitt Peak National Observatory. This camera helps observers monitor weather conditions and visibility, and confirm when it is safe to open the dome above the telescope. While the skies above Kitt Peak are very often clear, that is not always the case, as shown below in a series of camera images from the night of September 23, 2022. The night began with clouds and rain, and for over an hour flashes of lightning lit up both the sky and the camera view. Finally, toward the end of the night the storm gave way to thin clouds that allowed some stars to peek through. The final frame shows the Milky Way stretched across a mostly clear sky before the series starts over.

Credit: Luke Tyas

The all-sky camera can also capture celestial events, like the lunar eclipse that occurred on the night of November 8, 2022. DESI Lead Observer Luke Tyas created the timelapse below of the eclipse from 108 individual images from the all-sky camera. At first the full moon shines brightly just right of center, reflecting so much light into the sky that only a few bright stars are visible. As the eclipse begins the moon dims to a red dot scarcely brighter than the brightest stars, and the Milky Way can be seen following the eclipsed moon across the sky. As it nears the western horizon the moon emerges from the Earth’s eclipsing shadow, re-illuminating the telescope domes visible on the horizon and washing out the Milky Way.

Credit: Luke Tyas

David Sprayberry, NOIRLab

August 25, 2022

After the Contreras Fire swept over part of Kitt Peak in June, July was devoted to restoring electrical power and water service to the mountaintop. We also carefully assessed every structure for safety, possible fire damage, and the extent of cleaning required. Currently electricity at the summit is provided solely by back-up generators (more on that later). Because the water system lost pressure during the power outage, it was necessary to flush it with chlorine and then fresh water from the storage tanks, and then have the water tested to ensure it meets EPA standards for safe drinking water (it passed!). Work at the telescopes during July was limited to skeleton crews going up for specific tasks to prevent further issues with sensitive scientific equipment. At the Mayall, this included restarting the building’s cooling system, changing filters in the purified dry air system that purges the focal plane enclosure, and changing filters in the HVAC system serving the spectrograph clean room (fondly known as “the Shack”). We also performed regular inspections in all the NOIRLab domes for leaks and electrical faults.

Beginning August 1, telescope staff were allowed to resume almost normal on-site work. At the Mayall we spent the first week cleaning everything we could find a way to reach inside the dome, and any other areas potentially affected by ash or smoke intrusion. The interior of the dome wasn’t particularly dirty, but we cleaned everything thoroughly anyway. We also removed the protective tarp and plastic sheeting from the primary mirror and prime focus corrector, and made a preliminary inspection of the optics, which look fine. During the second week of August we caught up on overdue preventive maintenance of the telescope and dome, and worked to provide a low-bandwidth internet connection to the Mayall.

Road Conditions

Throughout August, road and weather conditions have significantly limited the time our team can spend on-site. The fire damaged guardrails along the road, and there has been a lot of erosion from burned areas washing across the road during heavy rains. This extraordinary erosion has clogged many of the drainages that cross the road, sometimes causing impassible mudflows. Additionally, the fire damaged several drainage culverts under the road. The Arizona Department of Transportation (ADOT) is making progress on repairs, but they are hampered by frequent heavy summer “monsoon” thunderstorms since late July. Finally, erosion is worsening the normal amount of rockfall along the road during storms. Combined with this year’s strong, frequent storms, we are encountering more rocks and boulders in the road.

Road restrictions are still in place. Repair work frequently requires closing both lanes, and so that we interfere as little as possible ADOT requires us to go up and down the mountain in one convoy at the same times every morning and afternoon, weather permitting. NOIRLab is also concerned for site safety due to the possibility of road closures caused by mudflows or rockfall. Whenever rain begins or is threatening, everyone on the site ceases work, makes their work site safe, and assembles for departure in an early convoy.

Electrical Power and Internet

The fire damaged between 10 and 20 utility poles that bring electricity and fiber optic data to the summit. Our utility provider, the Tohono O’odam Utility Authority (TOUA), is working hard to replace the damaged poles, but they face a number of obstacles. Many of the poles are in very remote locations. Other lines and poles elsewhere on Tribal lands were also damaged by storms. And like everyone else TOUA is grappling with supply chain disruptions. It will be at least several weeks until power is restored, and then fiber optic lines will be replaced.

The observatory is currently powered by on-site backup generators. One serves the Mayall and adjacent University of Arizona buildings, and another serves the rest of the mountaintop. This is expensive in terms of refueling needs, and leaves the site without back-up power sources if one of the generators were to fail. We are in the process of renting additional generators and connecting them to use as primary power sources. Once this is done the permanent back-up generators can return to back-up status.

Keeping the generators fueled is a challenge. Not all vendors will deliver diesel fuel to our remote site. Recently our normal supplier was unable to make our regular delivery when their truck broke down, and the Mayall generator ran out of fuel before an alternate supplier could reach the summit. Storms and road conditions further complicate the situation.

For temporary internet service DESI has obtained a Starlink system and service for the Mayall. The equipment is installed and connected to one network switch in the Mayall building, but that connection requires a standalone DNS server in the Mayall to allow normal access to the many other computers and switches in the building. DNS within NOIRLab is normally provided by one central server in Tucson, but that is obviously not an option right now, so work is underway to construct this standalone DNS system.

Optics Cleaning

We want to clean both the primary mirror and front surface of the prime focus corrector with carbon dioxide “snow”, and do a wet wash of the primary mirror. However, these operations require a relative humidity of < 50%, sustained through the day, to prevent condensation after the “snow” cleanings and promote drying after the wet wash. We haven’t seen a relative humidity reading that low since the August 1 return to the summit. DESI very much wants the optics cleaned before going back on-sky, as relative throughput measurements using the guide cameras are the first tests called for. We’re waiting for the first relatively dry day to seize an opportunity for optics cleaning.

Restarting DESI

Restart of the DESI equipment can’t really begin until some internet connectivity is restored and remote monitoring of the equipment’s health is possible. The expected order of events is to re-establish a fully-functioning network with internet connectivity; then bring the DESI computer cluster back on line; and then change filters inside the focal plane cooling system, restart focal plane cooling, and test basic focal plane and fiber view camera functions.

Once the focal plane guiders are running we can perform a simple throughput test if the optics have been cleaned and observing conditions are good. The next major phase will be restoring spectrograph operations. This involves restarting the cryostat control systems, pumping down cryostats, turning on the cryocoolers, and waiting for the CCDs to reach working temperature. The spectrograph restarts do not depend on weather conditions, but we do need a back-up electrical power supply or to know that one is on the way, so we don’t risk another unplanned warm-up were the back-up generator to fail.

We don’t yet have a firm timeline for restarting DESI, as so much depends on the weather and other things we have little or no ability to control. We’re doing the best we can, and will continue to post updates as developments warrant.

Anand Raichoor and Christophe Yeche

August 22, 2022

We are glad to announce the publication on arXiv on Friday, August 19, 2022 of eight papers from Year 1 Key Project 1 (Y1KP1). These papers describe:

• how the DESI Main Survey targets are selected, along with their photometric and main spectroscopic properties (MWS: Cooper et al. 2022, BGS: Hahn et al. 2022, LRG: Zhou et al. 2022, ELG: Raichoor et al. 2022; QSO: Chaussidon et al. 2022);
• the pipeline to process those targets for DESI observations (Myers et al. 2022);
• the building of redshift truth tables based on visual inspections (Lan et al. 2022, Alexander et al. 2022).

Targets are selected from the Legacy Survey DR9 photometric catalogs, which are derived from three optical imaging surveys in the grz-bands—primarily assembled for that exact purpose, completed with the near-infrared WISE and the Gaia data.

They summarize the long-term effort by many researchers from the DESI collaboration to design a key part of the DESI experiment. All the different algorithms were tested and optimized during the Survey Validation (SV). At the end of the first part of the SV, the team has provided a final version of the target selection for all the tracers. They were finally validated during the One-Percent Survey.

With this target selection work coming to an end, the effort is now focusing on participating in the preparation of the large-scale structure analysis of the first year of data.

A second batch of papers will be published on arXiv in the coming months, prior to the SV data release…so stay tuned!

Angela Berti, University of Utah

June 29, 2022

On Saturday, June 11 a lightning strike started a wildfire in the Arizona mountains less than ten miles from the Kitt Peak National Observatory (KPNO), where DESI observing is done with the Mayall 4-meter telescope. Besides the Mayall, Kitt Peak is home to over 20 telescopes and other buildings that support scientific observations on the mountain, including dormitories where staff sleep. All of these structures were potentially threatened by the wildfire.

By Tuesday, June 14 the wildfire (now called the Contreras fire) had spread to thousands of acres and was less than five miles from Kitt Peak. Around noon local time KPNO safety staff and the fire Incident Commander told everyone on site to immediately prepare to evacuate the mountain. Observing would have to be put on hold. The night of June 13 would be DESI’s final look at the sky for at least several weeks. Within hours a convoy was organized and began to bring those at the summit down the mountain to safety. A skeleton crew of four remained at the summit overnight for full-time fire watch.

On Wednesday, June 15 about 15 firefighters were at KPNO clearing defensive space around observatory buildings. Eight NOIRLab staff were also allowed to return for a few hours to protect critical equipment. This included taping plastic sheets and tarps over DESI’s prime focus corrector and the Mayall telescope’s primary mirror. These sensitive components could be damaged by smoke and ash should the fire get too close.

Nearly 200 firefighters were by then battling the Contreras fire, which was now only three miles away to the south. By the evening at least seven large air tankers were dropping fire retardant near KPNO.

The fire was just two miles away by the morning of June 16. Firefighting teams dropped over 100 loads of fire retardant along the perimeter of the observatory, and local news reported that the Contreras fire was the “#1 priority to wildland fire in the entire United States” due to the value of KPNO.

By early morning on Friday, June 17, the fire swept over the Southwest Ridge section of the observatory, home to MDM Observatory (two optical telescopes), the Arizona Radio Observatory, and the NRAO Very Long Baseline Array radio dish. KPNO webcams mounted on some of the telescopes stopped returning images not long after as the fire disrupted electricity and internet service on the mountain. DESI collaborators around the world could no longer monitor instruments remotely due to the loss of connectivity.

Good news finally came around midday on June 17 as light rain began to fall in nearby Tucson, Arizona. In the afternoon word came from two employees of NOIRLab who were on the mountain assisting firefighters with KPNO’s water system that no fire had reached the Mayall telescope. By the time the Contreras fire was 100% contained it had spread to nearly 30,000 acres. The fire destroyed four “non-scientific” structures, but none of the more than 20 telescopes atop Kitt Peak were burned!

On Tuesday, June 21, the DESI collaboration gathered in Berkeley, California for its first in-person meeting since 2019 due to the covid-19 pandemic. Many collaboration members who couldn’t be there in person joined remotely, and everyone expressed their gratitude for the incredible firefighters who saved DESI and KPNO from the Contreras fire.

Claire Lamman, Harvard University

February 28, 2022

We all know the recipe for a great scientific collaboration: novel technology, topical science goals, and cool stickers.

In November 2021 we achieved the final ingredient with the launch of our official DESI Swag Shop.

The DESI shop is hosted on the third-party site Redbubble. This is a place for independent artists to sell their designs on a variety of products—everything from t-shirts to shower curtains. While not specifically designed for organizations to distribute swag, it is a convenient way to make a large selection of items easily available to individuals around the world. We set our shop settings to 0% profit margin—which means DESI makes no money and the prices are as low as possible.

In honor of the opening of our online shop, I made a special DESI-gn that highlights both the science and instrumentation parts of our collaboration. “Dark Energy” is represented by a map of large-scale structure, and “Spectroscopic Instrument” is represented by our focal plane and its 5,000 robotic positioners.

Four months later, 486 items have been purchased from our shop. The most popular are:

1. DESI Logo sticker
2. DESI Plane sticker
3. DESI Logo magnet
4. DESI Plane t-shirt
5. M31 Focal Plane sticker

Beyond stickers and t-shirts the shop also has masks, mugs, puzzles, posters, and even socks. Here’s a look at how the popularity of these items compare:

Stats like this may be helpful for conference organizers deciding what types of merch products to use. The key takeaway here: never forget the stickers!