blog

Recovering from the Contreras Fire

Benjamin Weaver, NOIRLab

January 24, 2023

DESI collaborators have previously reported on the Contreras Fire and how it initially affected DESI operations along with an update as personnel were able to return to Kitt Peak National Observatory (KPNO). Even as we were cleaning up KPNO and the DESI instrument in August 2022, we knew that it would possibly be several months before the Tohono O’odham Utility Authority (TOUA) could replace the infrastructure bringing electrical power and line internet (a fiber-optic link) to KPNO. In fact, line internet was not restored until December 7, 2022, almost exactly six months after the start of the fire.

The story of restoring electrical power and the generators used while restoration was in progress is definitely worth telling, and far more lengthy and detailed than the story I’d like to tell now, recovering from the missing internet connection.

The DESI Data Management team, working closely with the on-site NOIRLab personnel, developed a plan to work around the missing internet, even as DESI returned to full-night operations. It wasn’t a new plan; in fact, it was a very old plan: “Sneakernet”. The idea is that a human being (presumably wearing sneakers) can hand-carry data on physical media from one location to another, and in certain circumstances, the effective bandwidth of that transfer can be much, much higher than the bandwidth of an internet connection or other communication channel. This is actually even more true now than it was in the past, because the storage density of physical media has experienced exponential growth similar to that of Moore’s law for CPU performance. However, while Moore’s law has been showing signs of reaching fundamental physical limits, storage density trends continue to rise.

With that in mind, we purchased six 2 TB SanDisk solid-state drives (SSDs), and a few fast USB cables. There is nothing exotic about the hardware; these items would be available in many consumer electronics or department stores. Then with the 6 disks initially carried to KPNO, we would place an entire night of data (or sometimes several nights) on a SSD. One of the personnel working at KPNO would be designated to carry the SSD to Tucson at the end of the work day. Then we would pick up the disk either in person or from a drop box and copy the data to NERSC where it could be processed with the usual DESI data pipeline. Since we purchased six SSDs, that meant we always had several SSDs on standby at KPNO in case an already-transferred SSD could not be brought back from Tucson to KPNO the next day. On weekends, no one would be available to carry the SSDs, so we simply had several nights on Monday evening’s SSD.  From the point of view of the pipeline, the only difference was that an entire night (or several nights) would arrive all at once, after a delay of 1 to 3 days, instead of the normal operations procedure of one exposure at a time, almost immediately after the exposure is completed.

The DESI instrument resumed taking data starting with night 20220825 (August 25, 2022). The first SSD was brought to Tucson on September 2, 2022. This process continued until line internet was restored on December 7, 2022. The original data transfer system was ready to go at that point and was reactivated almost immediately after line internet was restored. Thus the last night of data transferred by sneakernet was 20221206 (December 6, 2022). In total, 16,726 exposures over 100(!) nights were hand-carried to Tucson. This amounts to 6.1 TB of data.

The bandwidth of hand-carried SSDs is impressive, but so is the bandwidth of modern USB cables. After some experimentation, we discovered that we could transfer data directly from the external SSD to NERSC via Globus Online, with bandwidth performance indistinguishable from the case where the data were first copied to a workstation, then transferred via Globus. In other words, the limiting bandwidth was the internet itself between Tucson and NERSC, not the internal systems of the receiving workstation.

While we were transferring DESI data with a very old system (but with the latest SSDs), we were also experimenting with a new one: Starlink. In fact there were at least two separate Starlink installations active at KPNO during the recovery.  One, which we’ll call the “DESI Starlink”, was a strictly limited command and control channel for use by instrument experts. The DESI Starlink was not used for any significant data transfer. The other, which we’ll call the “NOIRLab Starlink” was used for more general access to all of KPNO. We did use the NOIRLab Starlink for some data transfer, with mixed results.

At least currently, Starlink works best with a point-to-point style networking connection, optimized for individuals or residences, much like a residential cable modem. Since the uplink bandwidth is currently a factor of 10-20 times slower than the downlink bandwidth, it is less practically suited as a two-way connection between two substantially-sized networks, such as between KPNO and the Tucson base facilities. This manifested as a significant, recurring routing problem. Although bandwidth was adequate for interactive use and for the transfer of small data files, the connection was frequently not working and the routing systems had to be monitored constantly. Nevertheless, we were able to use the NOIRLab Starlink to maintain an off-site copy of the DESI operations database. This copy would have been impractical to maintain using daily SSD transfers, because the copy is a real-time mirror of a live database; it is more of a stream than a set of files. There were occasions where the database mirror fell behind due to interruptions in the Starlink connection. In these cases files created by the database were transferred manually on the SSDs and then once in Tucson the data files were manually added to the database mirror.

It was interesting and a good experience to transfer data over sneakernet. It was also a lot of work and hours. I want to give credit to a big team who all contributed to this effort: Keith Blaine, Paul Demmer, Matthew Evatt, Steve Grandi, Bob Marshall, Rod Rutland, David Sprayberry, Christopher Stone, Bob Stupak (NOIRLab), Martin Landriau (LBNL), and Klaus Honscheid (OSU). Here’s one job we hope we never have to do again!

January 18, 2023

The all-sky camera on the Mayall telescope were the DESI instrument is located provides a view of the entire sky above Kitt Peak National Observatory. This camera helps observers monitor weather conditions and visibility, and confirm when it is safe to open the dome above the telescope. While the skies above Kitt Peak are very often clear, that is not always the case, as shown below in a series of camera images from the night of September 23, 2022. The night began with clouds and rain, and for over an hour flashes of lightning lit up both the sky and the camera view. Finally, toward the end of the night the storm gave way to thin clouds that allowed some stars to peek through. The final frame shows the Milky Way stretched across a mostly clear sky before the series starts over.

Credit: Luke Tyas

The all-sky camera can also capture celestial events, like the lunar eclipse that occurred on the night of November 8, 2022. DESI Lead Observer Luke Tyas created the timelapse below of the eclipse from 108 individual images from the all-sky camera. At first the full moon shines brightly just right of center, reflecting so much light into the sky that only a few bright stars are visible. As the eclipse begins the moon dims to a red dot scarcely brighter than the brightest stars, and the Milky Way can be seen following the eclipsed moon across the sky. As it nears the western horizon the moon emerges from the Earth’s eclipsing shadow, re-illuminating the telescope domes visible on the horizon and washing out the Milky Way.

Credit: Luke Tyas

David Sprayberry, NOIRLab

August 25, 2022

After the Contreras Fire swept over part of Kitt Peak in June, July was devoted to restoring electrical power and water service to the mountaintop. We also carefully assessed every structure for safety, possible fire damage, and the extent of cleaning required. Currently electricity at the summit is provided solely by back-up generators (more on that later). Because the water system lost pressure during the power outage, it was necessary to flush it with chlorine and then fresh water from the storage tanks, and then have the water tested to ensure it meets EPA standards for safe drinking water (it passed!). Work at the telescopes during July was limited to skeleton crews going up for specific tasks to prevent further issues with sensitive scientific equipment. At the Mayall, this included restarting the building’s cooling system, changing filters in the purified dry air system that purges the focal plane enclosure, and changing filters in the HVAC system serving the spectrograph clean room (fondly known as “the Shack”). We also performed regular inspections in all the NOIRLab domes for leaks and electrical faults.

Beginning August 1, telescope staff were allowed to resume almost normal on-site work. At the Mayall we spent the first week cleaning everything we could find a way to reach inside the dome, and any other areas potentially affected by ash or smoke intrusion. The interior of the dome wasn’t particularly dirty, but we cleaned everything thoroughly anyway. We also removed the protective tarp and plastic sheeting from the primary mirror and prime focus corrector, and made a preliminary inspection of the optics, which look fine. During the second week of August we caught up on overdue preventive maintenance of the telescope and dome, and worked to provide a low-bandwidth internet connection to the Mayall.

Road Conditions

Throughout August, road and weather conditions have significantly limited the time our team can spend on-site. The fire damaged guardrails along the road, and there has been a lot of erosion from burned areas washing across the road during heavy rains. This extraordinary erosion has clogged many of the drainages that cross the road, sometimes causing impassible mudflows. Additionally, the fire damaged several drainage culverts under the road. The Arizona Department of Transportation (ADOT) is making progress on repairs, but they are hampered by frequent heavy summer “monsoon” thunderstorms since late July. Finally, erosion is worsening the normal amount of rockfall along the road during storms. Combined with this year’s strong, frequent storms, we are encountering more rocks and boulders in the road.

Road restrictions are still in place. Repair work frequently requires closing both lanes, and so that we interfere as little as possible ADOT requires us to go up and down the mountain in one convoy at the same times every morning and afternoon, weather permitting. NOIRLab is also concerned for site safety due to the possibility of road closures caused by mudflows or rockfall. Whenever rain begins or is threatening, everyone on the site ceases work, makes their work site safe, and assembles for departure in an early convoy.

Electrical Power and Internet

The fire damaged between 10 and 20 utility poles that bring electricity and fiber optic data to the summit. Our utility provider, the Tohono O’odam Utility Authority (TOUA), is working hard to replace the damaged poles, but they face a number of obstacles. Many of the poles are in very remote locations. Other lines and poles elsewhere on Tribal lands were also damaged by storms. And like everyone else TOUA is grappling with supply chain disruptions. It will be at least several weeks until power is restored, and then fiber optic lines will be replaced.

The observatory is currently powered by on-site backup generators. One serves the Mayall and adjacent University of Arizona buildings, and another serves the rest of the mountaintop. This is expensive in terms of refueling needs, and leaves the site without back-up power sources if one of the generators were to fail. We are in the process of renting additional generators and connecting them to use as primary power sources. Once this is done the permanent back-up generators can return to back-up status.

Keeping the generators fueled is a challenge. Not all vendors will deliver diesel fuel to our remote site. Recently our normal supplier was unable to make our regular delivery when their truck broke down, and the Mayall generator ran out of fuel before an alternate supplier could reach the summit. Storms and road conditions further complicate the situation.

For temporary internet service DESI has obtained a Starlink system and service for the Mayall. The equipment is installed and connected to one network switch in the Mayall building, but that connection requires a standalone DNS server in the Mayall to allow normal access to the many other computers and switches in the building. DNS within NOIRLab is normally provided by one central server in Tucson, but that is obviously not an option right now, so work is underway to construct this standalone DNS system.

Optics Cleaning

We want to clean both the primary mirror and front surface of the prime focus corrector with carbon dioxide “snow”, and do a wet wash of the primary mirror. However, these operations require a relative humidity of < 50%, sustained through the day, to prevent condensation after the “snow” cleanings and promote drying after the wet wash. We haven’t seen a relative humidity reading that low since the August 1 return to the summit. DESI very much wants the optics cleaned before going back on-sky, as relative throughput measurements using the guide cameras are the first tests called for. We’re waiting for the first relatively dry day to seize an opportunity for optics cleaning.

Restarting DESI

Restart of the DESI equipment can’t really begin until some internet connectivity is restored and remote monitoring of the equipment’s health is possible. The expected order of events is to re-establish a fully-functioning network with internet connectivity; then bring the DESI computer cluster back on line; and then change filters inside the focal plane cooling system, restart focal plane cooling, and test basic focal plane and fiber view camera functions.

Once the focal plane guiders are running we can perform a simple throughput test if the optics have been cleaned and observing conditions are good. The next major phase will be restoring spectrograph operations. This involves restarting the cryostat control systems, pumping down cryostats, turning on the cryocoolers, and waiting for the CCDs to reach working temperature. The spectrograph restarts do not depend on weather conditions, but we do need a back-up electrical power supply or to know that one is on the way, so we don’t risk another unplanned warm-up were the back-up generator to fail.

We don’t yet have a firm timeline for restarting DESI, as so much depends on the weather and other things we have little or no ability to control. We’re doing the best we can, and will continue to post updates as developments warrant.

Anand Raichoor and Christophe Yeche

August 22, 2022

We are glad to announce the publication on arXiv on Friday, August 19, 2022 of eight papers from Year 1 Key Project 1 (Y1KP1). These papers describe:

• how the DESI Main Survey targets are selected, along with their photometric and main spectroscopic properties (MWS: Cooper et al. 2022, BGS: Hahn et al. 2022, LRG: Zhou et al. 2022, ELG: Raichoor et al. 2022; QSO: Chaussidon et al. 2022);
• the pipeline to process those targets for DESI observations (Myers et al. 2022);
• the building of redshift truth tables based on visual inspections (Lan et al. 2022, Alexander et al. 2022).

Targets are selected from the Legacy Survey DR9 photometric catalogs, which are derived from three optical imaging surveys in the grz-bands—primarily assembled for that exact purpose, completed with the near-infrared WISE and the Gaia data.

They summarize the long-term effort by many researchers from the DESI collaboration to design a key part of the DESI experiment. All the different algorithms were tested and optimized during the Survey Validation (SV). At the end of the first part of the SV, the team has provided a final version of the target selection for all the tracers. They were finally validated during the One-Percent Survey.

With this target selection work coming to an end, the effort is now focusing on participating in the preparation of the large-scale structure analysis of the first year of data.

A second batch of papers will be published on arXiv in the coming months, prior to the SV data release…so stay tuned!

Angela Berti, University of Utah

June 29, 2022

On Saturday, June 11 a lightning strike started a wildfire in the Arizona mountains less than ten miles from the Kitt Peak National Observatory (KPNO), where DESI observing is done with the Mayall 4-meter telescope. Besides the Mayall, Kitt Peak is home to over 20 telescopes and other buildings that support scientific observations on the mountain, including dormitories where staff sleep. All of these structures were potentially threatened by the wildfire.

By Tuesday, June 14 the wildfire (now called the Contreras fire) had spread to thousands of acres and was less than five miles from Kitt Peak. Around noon local time KPNO safety staff and the fire Incident Commander told everyone on site to immediately prepare to evacuate the mountain. Observing would have to be put on hold. The night of June 13 would be DESI’s final look at the sky for at least several weeks. Within hours a convoy was organized and began to bring those at the summit down the mountain to safety. A skeleton crew of four remained at the summit overnight for full-time fire watch.

On Wednesday, June 15 about 15 firefighters were at KPNO clearing defensive space around observatory buildings. Eight NOIRLab staff were also allowed to return for a few hours to protect critical equipment. This included taping plastic sheets and tarps over DESI’s prime focus corrector and the Mayall telescope’s primary mirror. These sensitive components could be damaged by smoke and ash should the fire get too close.

Nearly 200 firefighters were by then battling the Contreras fire, which was now only three miles away to the south. By the evening at least seven large air tankers were dropping fire retardant near KPNO.

The fire was just two miles away by the morning of June 16. Firefighting teams dropped over 100 loads of fire retardant along the perimeter of the observatory, and local news reported that the Contreras fire was the “#1 priority to wildland fire in the entire United States” due to the value of KPNO.

By early morning on Friday, June 17, the fire swept over the Southwest Ridge section of the observatory, home to MDM Observatory (two optical telescopes), the Arizona Radio Observatory, and the NRAO Very Long Baseline Array radio dish. KPNO webcams mounted on some of the telescopes stopped returning images not long after as the fire disrupted electricity and internet service on the mountain. DESI collaborators around the world could no longer monitor instruments remotely due to the loss of connectivity.

Good news finally came around midday on June 17 as light rain began to fall in nearby Tucson, Arizona. In the afternoon word came from two employees of NOIRLab who were on the mountain assisting firefighters with KPNO’s water system that no fire had reached the Mayall telescope. By the time the Contreras fire was 100% contained it had spread to nearly 30,000 acres. The fire destroyed four “non-scientific” structures, but none of the more than 20 telescopes atop Kitt Peak were burned!

On Tuesday, June 21, the DESI collaboration gathered in Berkeley, California for its first in-person meeting since 2019 due to the covid-19 pandemic. Many collaboration members who couldn’t be there in person joined remotely, and everyone expressed their gratitude for the incredible firefighters who saved DESI and KPNO from the Contreras fire.

Claire Lamman, Harvard University

February 28, 2022

We all know the recipe for a great scientific collaboration: novel technology, topical science goals, and cool stickers.

In November 2021 we achieved the final ingredient with the launch of our official DESI Swag Shop.

The DESI shop is hosted on the third-party site Redbubble. This is a place for independent artists to sell their designs on a variety of products—everything from t-shirts to shower curtains. While not specifically designed for organizations to distribute swag, it is a convenient way to make a large selection of items easily available to individuals around the world. We set our shop settings to 0% profit margin—which means DESI makes no money and the prices are as low as possible.

In honor of the opening of our online shop, I made a special DESI-gn that highlights both the science and instrumentation parts of our collaboration. “Dark Energy” is represented by a map of large-scale structure, and “Spectroscopic Instrument” is represented by our focal plane and its 5,000 robotic positioners.

Four months later, 486 items have been purchased from our shop. The most popular are:

1. DESI Logo sticker
2. DESI Plane sticker
3. DESI Logo magnet
4. DESI Plane t-shirt
5. M31 Focal Plane sticker

Beyond stickers and t-shirts the shop also has masks, mugs, puzzles, posters, and even socks. Here’s a look at how the popularity of these items compare:

Stats like this may be helpful for conference organizers deciding what types of merch products to use. The key takeaway here: never forget the stickers!

Edmond Chaussidon, CEA Saclay

January 12, 2022

Although DESI will be able to collect 5,000 spectra simultaneously, galaxies are still too numerous to be all observed during the next five years of observation. Target Selection (TS) is a crucial step to identify which objects to observe during the spectroscopic survey.

To constrain the cosmic expansion history through measurements of Baryon Acoustic Oscillations, DESI probes the matter in the Universe with four different tracers:

• Bright galaxies (BGS) in the redshift (z) range 0.05 < z < 0.4
• Luminous Red Galaxies (LRG) for 0.4 < z < 1.0
• Emission Line Galaxies (ELG) for 0.6 < z < 1.6
• Quasars (QSO) for 0.9 < z < 2.1

Quasars will also probe the Universe at higher redshift (z > 2.1) through Lyman-alpha forest studies measuring the light absorption by the gas in front of quasars.

Easy to say that we want to select different galaxy types, but how? The main information we need are the galaxy colors, which we can determine thanks to the measurements provided by photometric surveys. We used the catalogs of the DESI Legacy Imaging Surveys, a program conducted over more than 14,000 square degrees of sky from the Northern hemisphere, in three optical bands: g (in the blue-green), r (in the red), and z (in the red/near-infrared, and not to be confused with redshift!). The data were collected via three independent programs:

• The Beijing–Arizona Sky Survey (BASS) observed ∼5,100 square degrees of the North Galactic Cap (NGC) in g and r using the 2.3-meter Bok Telescope.
• The Mayall z-band Legacy Survey (MzLS) provided z-band observations over the same footprint as BASS using the 4-meter Mayall Telescope.
• The Dark Energy Camera Legacy Survey (DECaLS) was performed with the Dark Energy Camera (DECam) on the 4-meter Blanco Telescope. DECaLS observed the bulk of the Legacy Imaging Surveys footprint in g, r, and z.

These optical data were complemented by two infrared bands from the all-sky data of the WISE satellite, namely: W1 (3.4 μm) and W2 (4.6 μm).

The selection of the BGSs, LRGs and ELGs was mostly done based on conditions on the source colors. For the QSOs, however, which are harder to select because their colors are quite similar to those of the overwhelming stellar background, we developed a more complex approach based on machine learning. To further reduce the contamination of the sample from unwanted stars or galaxies, we applied an additional constraint on source magnitude. All these conditions were tuned on previous spectroscopic samples as those from the BOSS/eBOSS program. But of course, we also intensively tested them during the Survey Validation (SV) phases emulating a nominal observation with DESI. Voilà! Lo and behold, they all give very satisfying results.

DESI is now on sky, observing every night the quasars and the galaxies that we selected using these algorithms, on its way to build the largest 3D map of the Universe to date!

The Target Selection and Survey Validation steps are described in great detail in the upcoming eight scientific papers (one Overview, two for the SV, four for the TS and one for the Milky Way Survey). Pipeline and algorithms for data reductions and operations will be described in another set of five scientific papers later. So, stay tuned!

For the curious, below are more details on how we proceed for the four classes of targets, and some information about the density of objects we thus select.

---

Bright Galaxy Target Selection

Star–galaxy separation in BGS is performed using a G_Gaia – r_raw cut. This criterion exploits the fact that the Gaia magnitude is measured with an aperture of a space-based point-spread function (PSF) while the Legacy Imaging Surveys magnitude captures the light from the entire source. This cut separates point sources (stars) from extended sources (galaxies). This selection will observe ~850 targets per square degree. (See Hahn et al., in prep.)

---

Luminous Red Galaxy Target Selection

LRG selection is done by the cut (red line) in the (r – z) – (z – W1) space. This selection uses the W1 infrared band to separate galaxies (color points) from stars (grey points). The different colors show the redshift of the galaxies in the color–color space. This selection will observe ~615 targets per square degree. (See Zou et al., in prep.)

---

Emission Line Galaxy Target Selection

To avoid stellar contamination (black smooth line) in the target selection, ELG selection uses a cut in (r – z) – (g – r) space. The color histogram is the redshift distribution of ELGs in the color space. This selection will observe ~2,387 targets per square degree. (See Raichoor et al., in prep.)

---

Quasar Target Selection

Only point sources are considered during the selection and the target selection aims to separate quasars from stars. However, the separation between these two classes is less obvious than in the previous cases. A more sophisticated selection has to be applied. Quasars are selected via a Random Forest classification instead of the classical color cuts selection done in previous spectroscopic surveys. The idea of the selection is to separate quasars from stars based on the “infrared excess” of QSOs. The stellar locus is illustrated by the red line and the quasars by the blue/green/yellow points. This selection will observe ~308 targets per square degree (see Chaussidon et al., in prep.).

David Lee Summers, Kitt Peak National Observatory

November 17, 2021

Many people who hear the word “astronomer” imagine a scientist spending long nights in an observatory dome peering through a telescope’s eyepiece. In fact, when the Nicholas U. Mayall telescope was commissioned in 1973, astronomers could sit at the telescope’s focus to operate the cameras. In those early days, the telescope operator worked at an analog console in a control room inside the dome. Over the years those telescope controls migrated to computers and the astronomers joined the telescope operators in the control room.

The control room is a cozy space. It’s a long, narrow room. The old analog console dominates one wall. Desks and shelves line the other walls. The closest restroom is two floors below the control room and the control room didn’t even have a sink until one was installed circa 2010. A small microwave allowed the operator and observers to heat their night lunches. When planning for the DESI survey began, people soon realized this space was far too small for the number of scientists who would need to collaborate on a given night while commissioning the instrument. Even during regular observing, it was expected that three or four people would need to be in the control room at the same time.

A new, larger space within the Mayall building was identified to serve as the control room. A former recreation room on the so-called Utility floor was renovated into a modern control room in 2017. Even before DESI operations began, we moved into the new control room to finish the Mosaic Z-band Legacy Survey. The new control room allowed several scientists to collaborate during the night. There was plenty of desk space for computers and monitors to watch all the telescope and DESI functions. An adjoining room held a small kitchenette and a meeting room. What’s more, restrooms were only one room further on. The U-floor control room seemed a vast improvement as a practical workspace—at least until the unexpected happened and operations had to be stopped in March 2020 for the COVID-19 pandemic.

Although we paused on-sky DESI commissioning, much thought was given to safe operations. Clearly we couldn’t have several people working together in one enclosed room, especially in the days before a vaccine had been developed and before the effectiveness of masks had been demonstrated.

Before the DESI installation began, some astronomers who used Kitt Peak facilities already observed remotely from their home institutions. They could log in and control an instrument through the internet and communicate with the telescope operator through video conferencing software. This idea became the core for new, safe DESI operations. The telescope operator would return to the old control room next to the telescope. A lead DESI observer would work in the new U-floor control room. All other collaborators would work either from home or their home institutions and communicate with the team at the telescope over Zoom.

As an operator who has worked at Kitt Peak since the 1990s, it’s been gratifying to watch DESI’s development and see this next step in the Mayall Telescope’s illustrious history. It’s also been gratifying to see how elements of the telescope’s legacy along with new technology have kept us operating through these challenging times.

Adam Bolton, NOIRLab

November 1, 2021

DESI collaborator Dr. Frank Valdes, a scientist at NSF’s NOIRLab, has been honored with the 2021 ADASS Prize for an Outstanding Contribution to Astronomical Software. The award recognizes Frank and two other colleagues for their foundational roles in creating the Image Reduction and Analysis Facility (IRAF) software system. IRAF was launched over three decades ago as a pioneering project to create general-purpose software tools that would allow astronomers to translate the images taken by telescopes into calibrated data products suitable for quantitative scientific analysis. In the years since its initial release, IRAF has been used in nearly 25,000 scientific publications. Most significantly for DESI, all the images collected by the Legacy Surveys (DECaLS, MzLS, and BASS) to enable DESI’s spectroscopic target selection were processed through an IRAF-based pipeline created by Frank and operated by him and other NOIRLab staff members. The DESI Collaboration congratulates Frank on this well-deserved recognition!

John Della Costa, University of Florida

October 2, 2021

Astronomy is fun. But have you ever wanted to make it even better? Then why not try upgrading your 100 million dollar world class astronomical instrument! This may sound like an extremely difficult challenge that would require more than 30 people and many weeks of labor intensive work around the clock, and you’d be right! But with this handy DIY guide, you’ll be able to contribute in no time at all! So, follow along with this step-by-step tutorial and you’ll be observing the universe with your unrivaled, upgraded instrument faster than you can say, “Oh no, I only see 12 positioners but I plugged in 13. Please don’t make me take the whole petal apart again!”

Step 1: Get to the mountain
Remote mountaintops in dry locations are the best places to house telescopes. If you’re lucky, your amazing instrument will be located on the top of one in Southern Arizona. But first, you have to get to it! After your 4 hour plane ride in the lap of luxury, rent a vehicle and drive for 50 minutes until you get to the mountain road gate. Make sure to remember the combination for the lock (you still have the email with it, right?). Next, drive up the winding path to the summit. Try not to break any vertebrae as your neck becomes rubber from the occasional glimpses of the massive telescope your instrument desperately clings to. You’ve done it, you’re on the mountain!

Step 2: Get your safety briefing
Your safety briefing is one of the most important steps required for you to upgrade your instrument. When you’re on the mountain, there will likely be many creatures just waiting to kill you! So, be prepared, and never, ever, leave your food next to your bed as you sleep with the window open.

Step 2.5: Did you get food for your stay?
Did you really forget to get food for your 3 week stay on a mountain? Did you think you were going to subsist off of cacti and spiders? Now, drive back down the mountain, go to the grocery store another hour away, and drive back another hour again. Make sure you bought enough food to completely fill the freezer in the house so that you’ll have extra to throw out when you leave.

Step 3: Start disassembly of the first petal
By the time you get back to the dome after gluing your freezer shut, Petal 4 should be out of the telescope. But, before you start disassembly, make sure to test those pesky positioners with their mysterious ailment that nobody can understand. Now you’re ready to start disassembly! Just kidding, you have to take off the covers first. Make sure you use the correct tool or you might strip the… You stripped the screws, didn’t you? Well, that’s ok! Just make sure you have your hammer and drill close by so you can start going to town on your extremely fragile, multi-million dollar petal! But luckily, no other screws will be stripped on any other petal (foreshadowing). Now that you’ve spent many precious hours on this, you can start to disassemble the petal! Make sure as you pull out each positioner’s connector you really get your thumb and index finger in there. Don’t worry though, this definitely won’t leave you with nubs after doing it 5000 times… Wow, you’re fast! It only took you 3 hours of nonstop work! You should probably go home now and get some rest. Luckily your accommodations are only a 5 minute drive away (on the top of a mountain in complete darkness)!

Step 4: Start petal reassembly
Once the petal is completely taken apart, all 500 positioners are successfully unplugged, and all the transverse boards are out, you can get started on petal reassembly! Make sure not to snap any fibers, it’s not like they’re irreplaceable or anything. Finally, after two to three days of nonstop work, you’ll have your petal completely put back together. Now, just put it back in the telescope with that giant crane attached to the dome that lifts the 200 pound petal 30 feet into the air (seriously though, this part is terrifying to watch).

Step 5: Rinse and repeat!
Now all you have to do is repeat steps 3 and 4 seven more times (you’ll probably only be able to get 8 petals done during your 3 weeks on the mountain)!

Step 5.5: Take some time off
Make sure to take a day off here and there. Go for a hike, see some cacti (make sure to pick up their fruit with no hesitation, they can’t hurt you!), drive down the mountain and back up again, and enjoy the 110 degree weather (it’s a dry heat)!

Step 6: Make sure it’s not too easy
Be sure that Petal 6 has a lot of issues so that you have to take it apart and put it back together. Twice! But don’t worry, it’s great practice, and your thumbs will thank you later.

Step 7: Bask in the glory of an upgraded instrument
Now that you’ve completed the disassembly and reassembly of your world class multi-object spectrograph, you can start observing, again, and get back to deciphering the mysteries of the universe!

Step 7.5:
Wait, you’re telling me we have to test the instrument for weeks after reassembling it? Will we ever get to start observing again?

Well, while you wait, you can look back at the over 5000 pictures you took while you were on the mountain and think back at the amazing, once in a lifetime experience you had while helping to upgrade one of the best astronomical instruments on planet Earth, and remember all the skills you learned and all of the amazing people you met along the way.

Claire Poppett, Lawrence Berkeley National Laboratory

October 1, 2021

Anybody who has observed with the DESI instrument has experienced uncommunicable positioners on the focal plane that lead to frustrating pauses in the observing schedule whilst the lead observer or a focal plane expert work to bring these devices back. During the Tuscon summer monsoon season in 2021 a team of DESI collaboration members worked with NOIRLab staff to upgrade the DESI instrument and make these experiences a problem of the past.

To understand why these positioners become uncommunicable, we must first understand how they communicate.

The DESI positioners communicate via a Controller Area Network (CAN bus) and are grouped into patches of 25-75 positioners per CAN bus. Unfortunately, we have discovered that if a single positioner has a faulty communication signal it can affect the other positioners in that patch. The bigger issue here is that this loss in communications leads to a loss in their telemetry and the possibility that they may exceed our temperature limits without our knowledge. The collaboration agreed that this problem needed to be fixed and so a remediation plan was developed.

It was determined that the simplest way to be able to turn off a faulty positioner was to add a relay switch into the hardware but this required a major upgrade to the architecture of the focal plane. This was not possible without removing the focal plane from the telescope and so at the end of July, NOIRLab staff began the arduous task of getting the telescope into a state that made it possible to remove focal plane petals and install them onto work carts staged in temporary clean tents on the C-floor of the Mayall.

Once the petals were on the floor every single fiber and every single positioner wire had to be handled in order to remove the old communications boards from the petals in order to allow new boards with relay switches to be installed. These upgrades were performed on all 10 petals and included extensive testing to ensure that all devices now communicated as reliably as possible. The final petal was inserted back into the focal plane on August 25th and by lunchtime the next day we were able to declare that we had a healthy focal plane.

After only 10 weeks of downtime, the instrument was returned to service during the week of September 12th. Recalibration of the focal plane. Main Survey observations re-commenced on the evening of September 21. Despite the full moon, 60,000 Bright Galaxy Survey redshifts were measured that first night back!

All signs so far point to a truly upgraded DESI focal plane and we can look forward to many more years of DESI science!

Claire Lamman, Harvard University

August 20, 2021

For many, the term “explorers” brings to mind long ocean voyages, adventurers plunging into jungles, and traders discovering new connections between continents. Exploring, as a profession, became obsolete as there were simply no large, unexplored regions left and Earth became thoroughly mapped out.

However, there are still many uncharted regions beyond our planet. Hundreds of modern-day explorers are mapping out the galaxies around us, a craft that I like to think of as Cosmic Cartography. Fortunately for us, there’s not as much scurvy and snakes involved, but there is a massive effort behind today’s surveys and cosmology comes with its own set of adventures. 

DESI is creating the most detailed map yet of nearby galaxies. To celebrate the connection between early terrestrial and early cosmic explorers, I’ve made this DESI art in the style of early world maps. Here is a guide to some of the details:

1. A history of our universe. Since the big bang, our universe went through many changes as matter formed and eventually made stars, galaxies, and us. To understand how these changes happen, and how fast the universe is growing, we need to understand what it’s made of.
2. Pie chart of the universe. From our observations, we know that normal matter (the stuff that stars, dust, you, and I are made of) only accounts for about 5% of content in universe. 26% is dark matter- matter we cannot see but can study through how its gravity affects the normal matter and light around it. The other 69% is dark energy. As far as we know, about the only thing it has in common with dark matter is a spooky name. Dark energy is the mysterious force we credit with the rapid expansion of the cosmos. For more information see this blog post.
3. Time Map. Light takes time to travel. The light we see from nearby galaxies is relatively young, but from older galaxies it comes from a much older universe. Therefore, we can get a glimpse at what the universe looked like at different times by mapping it out near and far away. This is the map made by the Sloan Digital Sky Survey (SDSS) of galaxies around us. Earth is at the center. It looks like an hourglass because those are the galaxies in the part of the sky where SDSS was measuring. If you look closely, you may be able to see some structure – in places with the most data you can pick out a sponge or web pattern.
4. DESI. This is our logo. Our mission is to make a more detailed map so we can explore some of the remaining questions about dark energy and the evolution of the universe.
5. The DESI Footprint. This map shows the parts of the sky where DESI is looking. We call it the “footprint”. One chunk is south of the Milky Way (left) and one is north of it (right). DESI has already picked out the 30 million galaxies that we will measure. For more information see this blog post.
6. DESI’s home. DESI is located on a mountaintop in Arizona, Kitt Peak (left). It’s part of the biggest telescope on the mountain, the Mayall (right).
7. The Telescope. This is what lies under the dome – a massive structure supporting a 15-ton primary mirror, fine-tuned optics, a focal plane, and a separate room filled with spectrographs. More on all of this below.
8. How it works. This is a simplified diagram of the telescope. First, light hits the primary mirror. It then reflects up into a series of lenses before hitting the focal plane. The focal plane is a disk of 5,000 robotic positioners, each able to gather the light of their own galaxy. The light from each robot is then sent via fiber optic cables to spectrographs, which measure the light so we can get a spectra for each galaxy (11).
9. The focal plane. We often plot the values of each positioner, as they’re positioned on the focal plane. This is a drawing of what we saw when the telescope pointed at Andromeda. This galaxy takes up the perfect amount of sky so that we have a “picture” of it just by looking at positioner values. The other galaxies are far enough away that each gets its own positioner.
10. The positioners. Here is a diagram of what one of the robotic positioners looks like, with a drawing of one in the background.
11. Spectra. This is the type of measurement we want to get of every galaxy. Light from a galaxy contains many signals, coded in the amount of light that is present for different amounts of energy. When you plot the intensity of light as a function of energy (or wavelength) you see something like this. This is the spectra of a quasar, a brilliant type of galaxy with a very active black hole at its center. We expect to see certain peaks at certain energy levels. These shift based on the galaxy’s movement and allow us to measure its distance. For more information see this blog post.

More details

12. Draco. Star locations and art of the constellation Draco. You can’t have an old map without a serpent!
13. Tomog. Tomog is the Oʼodham word for the Milky Way. The Tohono O’odham people are native to Kitt Peak and the land surrounding it.
14. Cosmic Linguistics. These equations help astronomers describe and study cosmology. They express everything from how further objects look dimmer to how different components of the universe affect its evolution.
15. DESI Dog. Our beloved mascot, the DESI coyote, with the constellation Canis major, or “the big dog”.

Victoria Fawcett, Durham University

July 22, 2021

As the main DESI survey gets truly underway, I want to reflect on the amazing quality of the DESI quasar spectra observed so far. Quasars are incredibly luminous types of Active Galactic Nuclei (AGN), which consist of a supermassive black hole at the centre of galaxies, surrounded by a disc of matter that often outshines the entire galaxy. Quasars are one of the brightest objects in the Universe and are really important in all areas of Astronomy, especially Cosmology.

With over 30,000 quasar spectra observed so far, DESI is already pushing beyond what previous spectroscopic surveys have achieved. The image shows these ~30,000 spectra stacked in different bins of redshift, with the lowest redshift bin (z<0.4) corresponding to the objects closest to us. The dashed lines indicate the main emission lines in the spectra, each of which provide important information about the nature of the object: for example, the Oxygen lines can be used to study outflows close to the supermassive black hole. The clarity of the spectra and different lines really highlights the amazing quality of the data—the two red dotted lines show galaxy absorption lines, so we can even clearly see the affect of the surrounding galaxy!

Other exotic objects such as Broad Absorption Line Quasars (BALQSOs), systems known to host powerful outflows, have also been found within DESI (see image below). The dips to the left of the CIV, SiIV and Lya line are called BAL “troughs”, which deepen with increasing “balnicity index (BI)”; a measure of the strength of the trough. Studying these systems may be really important for understanding the processes that connect AGNs and their host galaxies.

DESI also pushes to fainter and more obscured systems which have been difficult to observe with shallower spectroscopic surveys. Quasars enshrouded by dust (“red quasars”) may represent an important phase in galaxy evolution so understanding their properties is crucial – these are the objects I am most interested in!

With millions of more quasar spectra to come, the future of quasar physics looks bright. I personally look forward to analysing the data and exploring all the weird and wonderful objects DESI has to offer.

David Schlegel, DESI Project Scientist

July 22, 2021

Only eight weeks into DESI’s five-year mission, a 3-dimensional map of the universe is starting to take shape.

Plotted below is a “pie diagram” slice through the universe, with earth in the lower left, looking out in the directions of the constellations Virgo, Serpens and Hercules to distances beyond 5 billion light years. As this video progresses, the vantage point sweeps through 20 degrees towards Bootes and Corona Borealis. Each point represents a DESI target. The nearest are the bright galaxy sample (BGS as white points), then the luminous red galaxies (LRGs in red), emission line galaxies (ELGs in green), and finally quasi-stellar objects (QSOs in blue). Each of these targets is composed of 100 billion to 1 trillion stars, although we plot each target only as a single point. Gravity has clustered the galaxies into structures called the “cosmic web”, with dense clusters, filaments and voids.

DESI shuts down today for summer maintenance and upgrades, timed to coincide with the monsoon season in Arizona. When observations re-start in September, five times more galaxies will be observed at each sky location, gaps in this map will be filled in, and the area surveyed will eventually grow to span most of the sky visible from the northern hemisphere.

Bela Abolfathi, University of California Irvine

June 1, 2021

On May 17, DESI officially began a 5-year survey of the cosmos to study dark energy and its role in the accelerated expansion of the universe. Over the course of its survey, DESI will collect spectra from over 30 million galaxies across 11 billion years, a feat that will result in the largest 3D map of the universe ever created.

Prof. David Kirkby at UC Irvine created an interactive 3D visualization that helps put into perspective exactly how ambitious of an undertaking this is. The visualization linked to the image below shows around 30,000 galaxies observed during the survey validation phase—less than 0.1% of the galaxies DESI will eventually catalog—jam-packed into the space behind your palm held at arm’s length.

Each of the four types of galaxies specifically targeted by DESI are denoted by a separate color and correspond to bright galaxies (BGS), luminous red galaxies (LRG), emission-line galaxies (ELG), and quasars (QSO). These types populate different redshift regimes, as seen from the progression of colors over the course of the movie. That’s not to say that types are confined to select regions of redshift-space. Rather, they are targeted in this way because, once they’ve been redshifted to our telescope, their unique signatures fall into the 360-980 nanometer window registered by our detectors. In the case of high-redshift quasars, these objects are selected because they help reveal the distribution of matter in the intergalactic medium via the shadows they cast along their journey.

Like the opening crawl at the beginning of every Star Wars movie, the visualization starts in the nearby universe and takes us back in time to distances far, far away. Exactly how do we determine these distances? We start with each galaxy’s redshift, which we can measure from its observed spectrum. Both the composition and curvature of the universe will determine how fast it’s expanding, and by extension any notion of distance. We therefore need to assume a particular cosmological model in order to then transform from redshift to comoving distance.

The visualization defaults to using the fiducial values of 31% matter, 69% dark energy, and zero curvature. Changing these parameters will have a noticeable effect on the distance traveled as well as the age of the universe.

One of DESI’s main cosmological probes is baryon acoustic oscillations (BAO), the imprints of relic sound waves from the early universe. BAO is a standard ruler used to measure the scale at which galaxies tend to cluster. The concentric circles emanating from the center of the 3D visualization show the comoving 150 Mpc BAO scale at which we expect galaxies to cluster slightly more often compared to a random distribution of galaxies.

Michael Levi, DESI Director

May 14, 2021

On the eve of the official start of the DESI science survey, I wanted to take a minute to reflect on how we arrived at this exciting moment. We’ve now proven that we can obtain 100,000 science-quality spectra in a single night, so we are ready! It has been quite a long and at times bumpy road to get here, but it has always been an honor to be part of this team. Of course, the ground was prepared by SDSS-BOSS, which established BAO as a precision probe of dark energy and created a desire to go beyond plug-plates. The seeds of this project germinated in 2008 out of the DOE/NASA JDEM space mission competition, when Saul Perlmutter and I were co-PI’s of the SNAP/JDEM proposal. The competing JDEM concepts led to the realization that a lot of the proposed BAO science could be performed from the ground, and this in turn led very quickly to a new concept for a ground-based spectroscopic survey. A rapid-fire set of ad hoc proposals and public presentations to DOE and NSF agency committees ensued as a core formed around this idea.

At the 2009 Particle Astrophysics Scientific Assessment Group (a subpanel of the NSF/DOE HEPAP Federal Advisory Committee), Nikhil Padmanabhan (then an LBNL Chamberlain Fellow), gave a presentation, which resulted in a recommendation to support the R&D. David Schlegel presented the BigBOSS concept to the 2010 Decadal Survey, which recommended mid-scale projects, explicitly calling out ground-based spectroscopic surveys as a compelling area of interest. In early 2011, two unsolicited proposals were submitted to DOE, for BigBOSS led by LBNL, and for DESpec a competing concept led by FNAL. This prompted the formation in mid-2011 of a dark energy community planning panel, chaired by Rocky Kolb, to advise the agency. The panel identified a wide-field spectroscopic survey as a key project that would meet the needs of the dark energy community. DOE agreed and, in the year following, approved “Mission Need” (aka CD-0) for a Mid-Scale Dark Energy Spectroscopic Instrument (MS-DESI) on Sept 18, 2012. 

DOE didn’t make a down-select between BigBOSS and DESpec. Instead, they assigned the management of the new MS-DESI project to Berkeley Lab on Dec 12, 2012 and charged us to form a new collaboration. I was appointed Project Director the next week and given the task to figure it out and come back with a plan. Fortunately, we had just received a $2.1M grant from the Gordon and Betty Moore Foundation for a spectroscopic survey (followed a bit later by a similar grant from the Heising-Simons Foundation) and this provided the financial resources for us to get started building the first spectrograph and initiating the long-lead acquisition of the corrector lenses.  At the same time, we began to hold collaboration meetings and created a “big tent” where all were welcome, and the DESI collaboration grew to include most of the institutions of the original proposals, plus many additional recruits.

A year later, in 2013, I presented the new plan and the newly formed science goals to the Particle Physics Project Prioritization Panel (P5), a subpanel of the NSF/DOE HEPAP Federal Advisory Committee. The P5 report, issued in mid-2014, recommended MS-DESI only in an optimistic budget scenario. Fortunately, budgets did align sufficiently for us to eke out a start in 2015 (albeit financially constrained) with CD-1 approval (site selection, and establishment of cost range) in March 2015 and with Congressional authorization in the appropriations language that year. DESI was then baselined in Sept 2015, construction started, and the rest is history as they say. Although there were plenty of remaining bumps in the road during construction, I will save that for another time.

It is a privilege for me to represent the collaboration and contribute to its continuing success, and I hope all our collaborators feel equally privileged. The DESI collaboration has come together in this common quest to understand dark energy and the mysteries of the accelerating Universe. Now the telescope and instrument are humming, twilight and science awaits.