Dark Energy Spectroscopic Instrument (DESI)

Contreras Fire Threatens DESI and Kitt Peak National Observatory

by

aberti

Angela Berti, University of Utah
June 29, 2022

On Saturday, June 11 a lightning strike started a wildfire in the Arizona mountains less than ten miles from the Kitt Peak National Observatory (KPNO), where DESI observing is done with the Mayall 4-meter telescope. Besides the Mayall, Kitt Peak is home to over 20 telescopes and other buildings that support scientific observations on the mountain, including dormitories where staff sleep. All of these structures were potentially threatened by the wildfire.

By Tuesday, June 14 the wildfire (now called the Contreras fire) had spread to thousands of acres and was less than five miles from Kitt Peak. Around noon local time KPNO safety staff and the fire Incident Commander told everyone on site to immediately prepare to evacuate the mountain. Observing would have to be put on hold. The night of June 13 would be DESI’s final look at the sky for at least several weeks. Within hours a convoy was organized and began to bring those at the summit down the mountain to safety. A skeleton crew of four remained at the summit overnight for full-time fire watch.

On Wednesday, June 15 about 15 firefighters were at KPNO clearing defensive space around observatory buildings. Eight NOIRLab staff were also allowed to return for a few hours to protect critical equipment. This included taping plastic sheets and tarps over DESI’s prime focus corrector and the Mayall telescope’s primary mirror. These sensitive components could be damaged by smoke and ash should the fire get too close.

Plastic sheeting taped over DESI’s prime focus corrector. Credit: Bob Stupak

Nearly 200 firefighters were by then battling the Contreras fire, which was now only three miles away to the south. By the evening at least seven large air tankers were dropping fire retardant near KPNO.

The fire was just two miles away by the morning of June 16. Firefighting teams dropped over 100 loads of fire retardant along the perimeter of the observatory, and local news reported that the Contreras fire was the “#1 priority to wildland fire in the entire United States” due to the value of KPNO.

By early morning on Friday, June 17, the fire swept over the Southwest Ridge section of the observatory, home to MDM Observatory (two optical telescopes), the Arizona Radio Observatory, and the NRAO Very Long Baseline Array radio dish. KPNO webcams mounted on some of the telescopes stopped returning images not long after as the fire disrupted electricity and internet service on the mountain. DESI collaborators around the world could no longer monitor instruments remotely due to the loss of connectivity.

The view of Kitt Peak National Observatory on the morning of June 17 from a camera mounted on the exterior of the Mayall telescope.
Credit: Clara Delabrouille

Good news finally came around midday on June 17 as light rain began to fall in nearby Tucson, Arizona. In the afternoon word came from two employees of NOIRLab who were on the mountain assisting firefighters with KPNO’s water system that no fire had reached the Mayall telescope. By the time the Contreras fire was 100% contained it had spread to nearly 30,000 acres. The fire destroyed four “non-scientific” structures, but none of the more than 20 telescopes atop Kitt Peak were burned!

On Tuesday, June 21, the DESI collaboration gathered in Berkeley, California for its first in-person meeting since 2019 due to the covid-19 pandemic. Many collaboration members who couldn’t be there in person joined remotely, and everyone expressed their gratitude for the incredible firefighters who saved DESI and KPNO from the Contreras fire.

David Schlegel, Lawrence Berkeley National Laboratory
May 14, 2022

In its first year of survey operations, DESI has dwarfed all prior redshifts surveys by mapping 12.8 million unique galaxies and quasars. One-quarter of those are at redshifts greater than 1.0.

Exactly one year ago, DESI began its five-year survey with over 100 DESI scientists on Zoom to witness the event. Although the instrument had been demonstrated to deliver science-quality data, this was not yet a smooth operation. At least a few scientists and engineers would join the observing staff each night to trouble-shoot problems with the telescope, the robotic focal plane, the guide cameras, the spectrographs, the CCD detectors, the cooling systems, or the control systems.

As the nights and weeks progressed, problems that would interrupt observations were identified and fixed. A 72-night shutdown starting July 11, 2021, serviced and replaced components of the robotic focal plane. The return-to-survey-operations on September 21 switched to our current nightly staffing model of an on-site Telescope Operator, an on-site Lead Observer, and two remote Support Observing Operators (who split the night). Only rarely are instrument experts still called in the middle of the night to trouble-shoot problems.

The observing efficiency has greatly improved from those first few nights of the survey. We routinely achieve 90% open-shutter time, defined as the time when the spectrograph shutters are open and collecting light from galaxies. The other 10% of the time is spent slewing the telescope to the next field, reading out the CCD detectors and re-configuring the focal plane (with various steps choreographed to occur concurrently). To date, the record open-shutter time was 10 hours 35 minutes on the long winter night of December 7, 2021.

During this first year, survey data has been collected on 242 nights. Of the remaining nights, 72 were for the maintenance shut-down, and only 51 were completely lost to weather or other engineering tasks. 2462 Main Survey DARK tiles and 2073 BRIGHT tiles have been observed. The raw data are transferred to the NERSC supercomputing center as they are collected, and are fully-reduced to calibrated spectra and redshifts by 10 AM the following morning. To date, the galaxy and quasar map consists of 12.8 million unique, reliable redshifts. In addition, 3.6 million unique stars have been observed. The number of redshifts is plotted as a function of time below.

The number of unique galaxies and quasars (top curve) and unique stars (bottom curve) with confidently-determined redshifts as a function of time. The first year of survey operations from May 14, 2021, through May 13, 2022, has delivered 12.8 million and 3.6 million such redshifts, respectively. (Anand Raichoor)

These next two figures show the distribution of observations on the sky. The survey began by observing non-overlapping tiles on the sky. Starting in September 2021, overlapping tiles were being observed that bring the typical number of visits of each patch of the sky to 5 for the DARK survey and 3 for the BRIGHT survey. Currently, prioritization is given to observing near the equator (declination 0), with diversions elsewhere to avoid pointing within 50 degrees of the moon or in the direction of the wind  windy nights. At Kitt Peak, the strongest winds are typically from the south, which has driven some observing to declinations > 32 degrees to avoid telescope wind shake. The DARK survey has observed 9500 square degrees (of a 14,000 square degree footprint) with at least one visit, with approximately 1300 square degrees fully completed with multiple visits.

Claire Lamman, Harvard University
February 28, 2022

We all know the recipe for a great scientific collaboration: novel technology, topical science goals, and cool stickers.

In November 2021 we achieved the final ingredient with the launch of our official DESI Swag Shop.

The DESI shop is hosted on the third-party site Redbubble. This is a place for independent artists to sell their designs on a variety of products—everything from t-shirts to shower curtains. While not specifically designed for organizations to distribute swag, it is a convenient way to make a large selection of items easily available to individuals around the world. We set our shop settings to 0% profit margin—which means DESI makes no money and the prices are as low as possible.

In honor of the opening of our online shop, I made a special DESI-gn that highlights both the science and instrumentation parts of our collaboration. “Dark Energy” is represented by a map of large-scale structure, and “Spectroscopic Instrument” is represented by our focal plane and its 5,000 robotic positioners.

Four months later, 486 items have been purchased from our shop. The most popular are:

  1. DESI Logo sticker
  2. DESI Plane sticker
  3. DESI Logo magnet
  4. DESI Plane t-shirt
  5. M31 Focal Plane sticker

Beyond stickers and t-shirts the shop also has masks, mugs, puzzles, posters, and even socks. Here’s a look at how the popularity of these items compare:

Stats like this may be helpful for conference organizers deciding what types of merch products to use. The key takeaway here: never forget the stickers!

New Scientist, 13 January 2022
Lawrence Berkeley National Laboratory, 13 January 2022
Sky & Telescope, 13 January 2022
Edmond Chaussidon, CEA Saclay
January 12, 2022

Although DESI will be able to collect 5,000 spectra simultaneously, galaxies are still too numerous to be all observed during the next five years of observation. Target Selection (TS) is a crucial step to identify which objects to observe during the spectroscopic survey.

To constrain the cosmic expansion history through measurements of Baryon Acoustic Oscillations, DESI probes the matter in the Universe with four different tracers:

  • Bright galaxies (BGS) in the redshift (z) range 0.05 < z < 0.4
  • Luminous Red Galaxies (LRG) for 0.4 < z < 1.0
  • Emission Line Galaxies (ELG) for 0.6 < z < 1.6
  • Quasars (QSO) for 0.9 < z < 2.1

Quasars will also probe the Universe at higher redshift (z > 2.1) through Lyman-alpha forest studies measuring the light absorption by the gas in front of quasars.

Easy to say that we want to select different galaxy types, but how? The main information we need are the galaxy colors, which we can determine thanks to the measurements provided by photometric surveys. We used the catalogs of the DESI Legacy Imaging Surveys, a program conducted over more than 14,000 square degrees of sky from the Northern hemisphere, in three optical bands: g (in the blue-green), r (in the red), and z (in the red/near-infrared, and not to be confused with redshift!). The data were collected via three independent programs:

  • The Beijing–Arizona Sky Survey (BASS) observed ∼5,100 square degrees of the North Galactic Cap (NGC) in g and r using the 2.3-meter Bok Telescope.
  • The Mayall z-band Legacy Survey (MzLS) provided z-band observations over the same footprint as BASS using the 4-meter Mayall Telescope.
  • The Dark Energy Camera Legacy Survey (DECaLS) was performed with the Dark Energy Camera (DECam) on the 4-meter Blanco Telescope. DECaLS observed the bulk of the Legacy Imaging Surveys footprint in g, r, and z.

These optical data were complemented by two infrared bands from the all-sky data of the WISE satellite, namely: W1 (3.4 μm) and W2 (4.6 μm).

The selection of the BGSs, LRGs and ELGs was mostly done based on conditions on the source colors. For the QSOs, however, which are harder to select because their colors are quite similar to those of the overwhelming stellar background, we developed a more complex approach based on machine learning. To further reduce the contamination of the sample from unwanted stars or galaxies, we applied an additional constraint on source magnitude. All these conditions were tuned on previous spectroscopic samples as those from the BOSS/eBOSS program. But of course, we also intensively tested them during the Survey Validation (SV) phases emulating a nominal observation with DESI. Voilà! Lo and behold, they all give very satisfying results.

DESI is now on sky, observing every night the quasars and the galaxies that we selected using these algorithms, on its way to build the largest 3D map of the Universe to date!

The Target Selection and Survey Validation steps are described in great detail in the upcoming eight scientific papers (one Overview, two for the SV, four for the TS and one for the Milky Way Survey). Pipeline and algorithms for data reductions and operations will be described in another set of five scientific papers later. So, stay tuned!

For the curious, below are more details on how we proceed for the four classes of targets, and some information about the density of objects we thus select.

Bright Galaxy Target Selection

Star–galaxy separation in BGS is performed using a GGaia – rraw cut. This criterion exploits the fact that the Gaia magnitude is measured with an aperture of a space-based point-spread function (PSF) while the Legacy Imaging Surveys magnitude captures the light from the entire source. This cut separates point sources (stars) from extended sources (galaxies). This selection will observe ~850 targets per square degree. (See Hahn et al., in prep.)

Luminous Red Galaxy Target Selection

LRG selection is done by the cut (red line) in the (r – z) – (z – W1) space. This selection uses the W1 infrared band to separate galaxies (color points) from stars (grey points). The different colors show the redshift of the galaxies in the color–color space. This selection will observe ~615 targets per square degree. (See Zou et al., in prep.)

Emission Line Galaxy Target Selection

To avoid stellar contamination (black smooth line) in the target selection, ELG selection uses a cut in (r – z) – (g – r) space. The color histogram is the redshift distribution of ELGs in the color space. This selection will observe ~2,387 targets per square degree. (See Raichoor et al., in prep.)

Quasar Target Selection

Only point sources are considered during the selection and the target selection aims to separate quasars from stars. However, the separation between these two classes is less obvious than in the previous cases. A more sophisticated selection has to be applied. Quasars are selected via a Random Forest classification instead of the classical color cuts selection done in previous spectroscopic surveys. The idea of the selection is to separate quasars from stars based on the “infrared excess” of QSOs. The stellar locus is illustrated by the red line and the quasars by the blue/green/yellow points. This selection will observe ~308 targets per square degree (see Chaussidon et al., in prep.).

David Lee Summers, Kitt Peak National Observatory
November 17, 2021

Many people who hear the word “astronomer” imagine a scientist spending long nights in an observatory dome peering through a telescope’s eyepiece. In fact, when the Nicholas U. Mayall telescope was commissioned in 1973, astronomers could sit at the telescope’s focus to operate the cameras. In those early days, the telescope operator worked at an analog console in a control room inside the dome. Over the years those telescope controls migrated to computers and the astronomers joined the telescope operators in the control room.

The original Mayall control room.

The control room is a cozy space. It’s a long, narrow room. The old analog console dominates one wall. Desks and shelves line the other walls. The closest restroom is two floors below the control room and the control room didn’t even have a sink until one was installed circa 2010. A small microwave allowed the operator and observers to heat their night lunches. When planning for the DESI survey began, people soon realized this space was far too small for the number of scientists who would need to collaborate on a given night while commissioning the instrument. Even during regular observing, it was expected that three or four people would need to be in the control room at the same time.

A new, larger space within the Mayall building was identified to serve as the control room. A former recreation room on the so-called Utility floor was renovated into a modern control room in 2017. Even before DESI operations began, we moved into the new control room to finish the Mosaic Z-band Legacy Survey. The new control room allowed several scientists to collaborate during the night. There was plenty of desk space for computers and monitors to watch all the telescope and DESI functions. An adjoining room held a small kitchenette and a meeting room. What’s more, restrooms were only one room further on. The U-floor control room seemed a vast improvement as a practical workspace—at least until the unexpected happened and operations had to be stopped in March 2020 for the COVID-19 pandemic.

The utility floor Mayall control room, circa 2018.

Although we paused on-sky DESI commissioning, much thought was given to safe operations. Clearly we couldn’t have several people working together in one enclosed room, especially in the days before a vaccine had been developed and before the effectiveness of masks had been demonstrated.

Before the DESI installation began, some astronomers who used Kitt Peak facilities already observed remotely from their home institutions. They could log in and control an instrument through the internet and communicate with the telescope operator through video conferencing software. This idea became the core for new, safe DESI operations. The telescope operator would return to the old control room next to the telescope. A lead DESI observer would work in the new U-floor control room. All other collaborators would work either from home or their home institutions and communicate with the team at the telescope over Zoom.

As an operator who has worked at Kitt Peak since the 1990s, it’s been gratifying to watch DESI’s development and see this next step in the Mayall Telescope’s illustrious history. It’s also been gratifying to see how elements of the telescope’s legacy along with new technology have kept us operating through these challenging times.

Adam Bolton, NOIRLab
November 1, 2021

DESI collaborator Dr. Frank Valdes, a scientist at NSF’s NOIRLab, has been honored with the 2021 ADASS Prize for an Outstanding Contribution to Astronomical Software. The award recognizes Frank and two other colleagues for their foundational roles in creating the Image Reduction and Analysis Facility (IRAF) software system. IRAF was launched over three decades ago as a pioneering project to create general-purpose software tools that would allow astronomers to translate the images taken by telescopes into calibrated data products suitable for quantitative scientific analysis. In the years since its initial release, IRAF has been used in nearly 25,000 scientific publications. Most significantly for DESI, all the images collected by the Legacy Surveys (DECaLS, MzLS, and BASS) to enable DESI’s spectroscopic target selection were processed through an IRAF-based pipeline created by Frank and operated by him and other NOIRLab staff members. The DESI Collaboration congratulates Frank on this well-deserved recognition!

This mosaic of the standard IRAF test image (M51, the Whirlpool Galaxy), found in every IRAF installation, is composed of plots and images taken from the more than 25,000 refereed papers published since 1986 that mention using IRAF.
Lawrence Berkeley National Laboratory, 28 October 2021
David Schlegel, Lawrence Berkeley National Laboratory
October 13, 2021

DESI has broken its own records, mapping the three-dimensional location of 100,000 distant galaxies and quasars in the single night of October 8, 2021.

The DESI Main Survey started on the evening of May 14, 2021. Nearly 100 scientists joined the operations team on Zoom that night to witness the beginning of the survey. It wasn’t a perfect night, as the “seeing” (blurring of the sky from atmospheric turbulence) was worse than typical, and some time was lost to instrument communication errors. Even still, eight dark-time tiles and ten bright-time tiles were observed, which was enough to set a record (for any telescope) of mapping 60,000 galaxies in a single night.

The operational efficiency of DESI has been improved since that first night. One important aspect of this is the “robustness” of the complex instrument and control systems. There are 5000 fiber robots on the top end of the telescope commanded to move every 15 minutes, 6 guide cameras, 4 optical wavefront sensors, 30 cryogenic cameras taking data in the spectrograph room, a large assortment of sensors, and dozens of computers to control all these systems. It’s not uncommon to have minor, non-critical faults somewhere in the system, and those faults have to be trapped rather than bring the whole system to a halt.

Operational efficiency also relies upon a careful choreography of reconfiguring the instrument between observations: the telescope is slewed to its next sky location, the focal plane is re-focused as the 375 tons of moving weight slightly sags under gravity, the 5000 fiber robots are re-positioned to point to the locations of unobserved galaxies, and the previous spectroscopic exposure is read out. The inter-exposure time for this dance of activities that was typically 3 minutes during the first night of the survey has been reduced to 2 min 15 sec. For this particular record-setting night, the open-shutter efficiency was 87%, meaning that’s the fraction of time that the shutters on the spectrographs were open and integrating on light from distant galaxies.

Finally, the weather needs to cooperate! The night of October 8th was a fully “dark” night, meaning that the moon was below the horizon. The skies were clear (“photometric”) for all but a few minutes when a thin cloud layer blew through. And the “seeing” was excellent, such that the blurring of the atmosphere was less than 1 arcsecond for most of the night. The sky wasn’t quite as dark as the best nights, partly because we were observing near the ecliptic plane where dust particles in the solar system reflect small amounts of light from the sun. In combination, the weather was sufficiently good that we were running at a survey speed of 120%—meaning 20% faster than expected under nominal, good conditions.

The night sky as seen from the Kitt Peak all-sky camera on the night of October 8, 2021. It was a dark, clear night, with the Milky Way passing overhead early in the night.

How many galaxies were mapped in this one night? We completed observations on 29 dark-time tiles (with one tile completing observations from a previous night). Each of those tiles included approximately 4200 high-redshift targets (galaxies or quasars), with the remaining 800 fibers assigned to calibration targets. Most of those galaxies were “redshifted” (mapped) in a single observation. These data were analyzed by 10 am the following morning on NERSC (the National Energy Research Scientific Computing Center) to confirm reliable redshifts for 98,954 galaxies and quasars. Additionally, two bright-time tiles observed during evening twilight measured redshifts for 4225 low-redshift galaxies, bringing the total haul to 103,179 galaxies and quasars.

Will this record be broken in the future? Most assuredly, if only because the nights are getting longer. This record-setting night had 9 hours 22 min of time between astronomical twilight (when the sky is darkest with the sun at least 18 degrees below the horizon). The best nights for setting records will be the new moons closest to the winter solstice. This year, that will be the nights of December 3 (10 hours, 57 minutes) and January 1, 2022 (10 hours, 59 minutes).

Congratulations to the observing team for the night of October 8, 2021: Liz Buckley-Geer (Fermilab, Lead Observer), Abby Bault (University of California Irvine, Support Observer), Mike Wang (University of Edinburgh, Support Observer) and Thaxton Smith (NOIRLab, Observing Assistant).

Christoph Saulder, Postdoc at KASI
October 11, 2021

While the main goal of DESI is to obtain spectroscopic redshifts for millions of galaxies to map the expansion history of the universe, there are also secondary targeting programs that supplement the main DESI survey and its science goals. The DESI peculiar velocity survey is one of these programs and it aims to improve the measurements of cosmological parameters in the local universe.

And yet it moves…
Even when accounting for the expansion of the universe (“Hubble flow”), galaxies do not sit around idly, but move around due to their mutual gravitational interactions. These proper motions, which are refereed to as peculiar velocities in our context, encode information about distribution of matter. By default, DESI measures the redshifts of galaxies, which are a combination of the cosmological redshifts caused by the expansion of the universe and the redshifts caused by peculiar motions. As the cosmological redshifts correlate with the distances to these galaxies, one can use redshift-independent distance indicators in combination with the redshift measurements to obtain the (radial) peculiar velocities of these galaxies. As we want to measure the peculiar velocities for a large number of sample, we cannot wait for a rare supernova type Ia to go off in each of them and most of them are too far away for us to resolve any Cepheids variable stars. We therefore have to rely on less precise, but more easily accessible redshift-independent distance indicators, such as galaxy scaling relations, in particular the fundamental plane and the Tully-Fisher relation. These allow us to measure the distances, and thereby the peculiar velocities, to a large number of galaxies in the local universe (redshift of less than 0.1).

Still waters run deep…
Which scaling relation a galaxy obeys depends on its morphological type. Galaxies with essentially no cold gas and therefore no ongoing star-formation appear as the relatively boring looking elliptical galaxies. These quiescent galaxies follow the fundamental plane, which relates the internal kinematics of these galaxies express as the central velocity dispersion to their surface brightness and physical size. By comparing the physical size predicted by this relation to its angular size on the sky, we are able to obtain the distance to such a galaxy by using simple trigonometry. While measurements of the angular size and surface brightness can be done on the already collected data of the photometric DESI Legacy Imaging Survey, we will need spectroscopic data for the central velocity dispersion. The central velocity dispersion quantifies the statistical dispersion of the motion of stars in the center of a galaxy.

The impact of the central velocity dispersion σ0 on the shape of two spectral lines. (credit: ChangHoon Hahn)

In DESI, fibers collect spectra from the innermost parts of these galaxies. In these spectra, the central velocity dispersion manifests as broadening of the spectral lines. However, to reliably recover these features, one needs spectroscopic data of excellent quality. Luckily most of our targets are bright galaxies, which are masked areas that within there are no high redshift galaxies targeted in DESI during dark time (best observing conditions). This means our additional observations have only little competition and over the course of the DESI survey, we will be able to gather data we need without interfering with the main DESI observations.

Round and Round it Goes…
The maximum rotation velocity of spiral galaxies is tightly linked to its absolute luminosity via the so-called Tully-Fisher relation. By comparing the predicted absolute luminosity to the observed apparent luminosity from DESI Legacy Imaging Survey, one can derive the distance to these galaxies. The maximum rotation velocity can be measured by placing fibers along the semi-major axis of the galaxy and just comparing the measured redshifts at these locations. As our targets are big and bright galaxies, the geometry of DESI often even requires fibers to placed somewhere on these galaxies, so we made sure that they will end up on the best possible location to maximize our science output.

These figures illustrate how the additional data about peculiar velocities can improve the strength of DESI cosmological measurements at low redshifts. (credit: Cullan Howlett)

The answer to life, the universe, and everything
While it seems to be simple, getting from redshifts and redshift-independent distance measurements to peculiar motions, it requires a lot of careful modeling of all kinds of biases and selection effects. In the end, we want to obtain solid measurements for cosmological parameters from the data. Peculiar velocities can supplement the measurements on the growth of structure that are usually from redshift space distortions and we can thereby can get tighter constraints on the cosmological parameter f σ8 , also refereed to as the growth rate (of structure). Its evolution over time and its scale-dependence can used to test for possible deviations from General Relativity. Additionally, by using galaxy scaling relations as distance indicators, we are able to collect data about the kinematics of a large sample of galaxies that will help us to gain better insights into the dark matter halo relation.

Step by step…
We were able to obtain our first measurements using the survey validation data of DESI, in particular the 1% survey. These first results helped us to test and improve our target selection methods, which are now applied to the ongoing main DESI survey. We are currently preparing several publications, which are to be released along with the first public release of DESI data, that will present the details of our target selection methods and the first results from the fundamental plane and Tully-Fisher relation. While data collected so far is sufficient for the basic calibrations of our methods, it will still take years until the survey is completed and we have sufficient data for strong scientific results and the chance of discovering any new physics beyond our current standard model.

John Della Costa, University of Florida
October 2, 2021

Astronomy is fun. But have you ever wanted to make it even better? Then why not try upgrading your 100 million dollar world class astronomical instrument! This may sound like an extremely difficult challenge that would require more than 30 people and many weeks of labor intensive work around the clock, and you’d be right! But with this handy DIY guide, you’ll be able to contribute in no time at all! So, follow along with this step-by-step tutorial and you’ll be observing the universe with your unrivaled, upgraded instrument faster than you can say, “Oh no, I only see 12 positioners but I plugged in 13. Please don’t make me take the whole petal apart again!”

Working carefully with fibers and electronic connectors.

Step 1: Get to the mountain
Remote mountaintops in dry locations are the best places to house telescopes. If you’re lucky, your amazing instrument will be located on the top of one in Southern Arizona. But first, you have to get to it! After your 4 hour plane ride in the lap of luxury, rent a vehicle and drive for 50 minutes until you get to the mountain road gate. Make sure to remember the combination for the lock (you still have the email with it, right?). Next, drive up the winding path to the summit. Try not to break any vertebrae as your neck becomes rubber from the occasional glimpses of the massive telescope your instrument desperately clings to. You’ve done it, you’re on the mountain!

Step 2: Get your safety briefing
Your safety briefing is one of the most important steps required for you to upgrade your instrument. When you’re on the mountain, there will likely be many creatures just waiting to kill you! So, be prepared, and never, ever, leave your food next to your bed as you sleep with the window open.

Step 2.5: Did you get food for your stay?
Did you really forget to get food for your 3 week stay on a mountain? Did you think you were going to subsist off of cacti and spiders? Now, drive back down the mountain, go to the grocery store another hour away, and drive back another hour again. Make sure you bought enough food to completely fill the freezer in the house so that you’ll have extra to throw out when you leave.

Step 3: Start disassembly of the first petal
By the time you get back to the dome after gluing your freezer shut, Petal 4 should be out of the telescope. But, before you start disassembly, make sure to test those pesky positioners with their mysterious ailment that nobody can understand. Now you’re ready to start disassembly! Just kidding, you have to take off the covers first. Make sure you use the correct tool or you might strip the… You stripped the screws, didn’t you? Well, that’s ok! Just make sure you have your hammer and drill close by so you can start going to town on your extremely fragile, multi-million dollar petal! But luckily, no other screws will be stripped on any other petal (foreshadowing). Now that you’ve spent many precious hours on this, you can start to disassemble the petal! Make sure as you pull out each positioner’s connector you really get your thumb and index finger in there. Don’t worry though, this definitely won’t leave you with nubs after doing it 5000 times… Wow, you’re fast! It only took you 3 hours of nonstop work! You should probably go home now and get some rest. Luckily your accommodations are only a 5 minute drive away (on the top of a mountain in complete darkness)!

Step 4: Start petal reassembly
Once the petal is completely taken apart, all 500 positioners are successfully unplugged, and all the transverse boards are out, you can get started on petal reassembly! Make sure not to snap any fibers, it’s not like they’re irreplaceable or anything. Finally, after two to three days of nonstop work, you’ll have your petal completely put back together. Now, just put it back in the telescope with that giant crane attached to the dome that lifts the 200 pound petal 30 feet into the air (seriously though, this part is terrifying to watch).

Step 5: Rinse and repeat!
Now all you have to do is repeat steps 3 and 4 seven more times (you’ll probably only be able to get 8 petals done during your 3 weeks on the mountain)!

Step 5.5: Take some time off
Make sure to take a day off here and there. Go for a hike, see some cacti (make sure to pick up their fruit with no hesitation, they can’t hurt you!), drive down the mountain and back up again, and enjoy the 110 degree weather (it’s a dry heat)!

John on the mountain with the Mayall Telescope in the background.

Step 6: Make sure it’s not too easy
Be sure that Petal 6 has a lot of issues so that you have to take it apart and put it back together. Twice! But don’t worry, it’s great practice, and your thumbs will thank you later.

Step 7: Bask in the glory of an upgraded instrument
Now that you’ve completed the disassembly and reassembly of your world class multi-object spectrograph, you can start observing, again, and get back to deciphering the mysteries of the universe!

Step 7.5:
Wait, you’re telling me we have to test the instrument for weeks after reassembling it? Will we ever get to start observing again?

Well, while you wait, you can look back at the over 5000 pictures you took while you were on the mountain and think back at the amazing, once in a lifetime experience you had while helping to upgrade one of the best astronomical instruments on planet Earth, and remember all the skills you learned and all of the amazing people you met along the way.

Claire Poppett, Lawrence Berkeley National Laboratory
October 1, 2021

Anybody who has observed with the DESI instrument has experienced uncommunicable positioners on the focal plane that lead to frustrating pauses in the observing schedule whilst the lead observer or a focal plane expert work to bring these devices back. During the Tuscon summer monsoon season in 2021 a team of DESI collaboration members worked with NOIRLab staff to upgrade the DESI instrument and make these experiences a problem of the past.

A technician carefully removes fibers (blue) and electronic connectors (rainbow) from electrical and mechanical supports.

To understand why these positioners become uncommunicable, we must first understand how they communicate.

The DESI positioners communicate via a Controller Area Network (CAN bus) and are grouped into patches of 25-75 positioners per CAN bus. Unfortunately, we have discovered that if a single positioner has a faulty communication signal it can affect the other positioners in that patch. The bigger issue here is that this loss in communications leads to a loss in their telemetry and the possibility that they may exceed our temperature limits without our knowledge. The collaboration agreed that this problem needed to be fixed and so a remediation plan was developed.

It was determined that the simplest way to be able to turn off a faulty positioner was to add a relay switch into the hardware but this required a major upgrade to the architecture of the focal plane. This was not possible without removing the focal plane from the telescope and so at the end of July, NOIRLab staff began the arduous task of getting the telescope into a state that made it possible to remove focal plane petals and install them onto work carts staged in temporary clean tents on the C-floor of the Mayall.

One of 10 petals being removed from the back of the focal plane with the telescope secured at the South-East platform.

Once the petals were on the floor every single fiber and every single positioner wire had to be handled in order to remove the old communications boards from the petals in order to allow new boards with relay switches to be installed. These upgrades were performed on all 10 petals and included extensive testing to ensure that all devices now communicated as reliably as possible. The final petal was inserted back into the focal plane on August 25th and by lunchtime the next day we were able to declare that we had a healthy focal plane.

After only 10 weeks of downtime, the instrument was returned to service during the week of September 12th. Recalibration of the focal plane. Main Survey observations re-commenced on the evening of September 21. Despite the full moon, 60,000 Bright Galaxy Survey redshifts were measured that first night back!

All signs so far point to a truly upgraded DESI focal plane and we can look forward to many more years of DESI science!

What is your role in the DESI project?
I’m a post-doc mainly involved in GQC working group.

Where were you born? Where do you live now? 
I was born in Seoul, Korea and currently living in Shanghai, China for my post-doc position at SJTU.

What would you say is the most interesting or exciting thing about DESI?
One of the most interesting thing about DESI for me is to experience the pipeline for processing spectroscopic data. Before joining DESI, ‘data’ were just some binary files to me consisting of many columns including RA, DEC, z, and some weights. But, after joining DESI, I had many chances to learn about how raw data are processed in the pipelines before arriving at my hands. For example, participating in visual inspection and working on fiber assignment are the ones where I can experience some part of the raw data process.

Any advice for aspiring scientists?
Either failure or success, it can be a stepping stone for the next step.

Finally, what do you do for fun?
Reading books. Rather than reading fast, I prefer slow reading and enjoying each line, which makes me feel like having conversation with author. And, recently, (I’m not sure if I should say “because of“ or “thanks to” COVID19, but anyway) I started a couple of online book clubs with some old friends. After reading one chapter of the same book every week or every two weeks, we regularly meet online and discuss anything related to the chapter. You may want to try this as well. It is really fun!