• Skip to main content
  • Skip to primary sidebar
  • Skip to footer

Dark Energy Spectroscopic Instrument (DESI)

  • / science /
    • science overview
    • cosmology and dark energy
    • redshifts and distance
    • mapping the universe
    • the DESI science mission
    • the DESI survey
    • imaging surveys
  • / instrument /
    • instrument overview
    • telescope
      • tohono o’odham
    • corrector
    • focal plane system
    • fiber system
    • spectrograph
    • instrument control system
    • data systems
    • bringing DESI to life
      • commissioning Instrument
      • protoDESI
  • / collaboration /
    • DESI team
    • DESI builders
    • collaborating institutions
    • sponsors
    • code of conduct
    • vendors
    • collaboration policies
  • / press /
    • announcements
    • in the news
    • press releases
    • tweets by desisurvey
    • blog
    • acknowledgments
  • / galleries /
    • videos
    • image gallery
  • / for scientists /
    • data releases
    • instrument design
    • imaging data
    • target selection and survey validation
    • theory and simulations
    • other DESI science
    • key publications
    • all DESI papers
    • team login
    • request a DESI speaker
    • internal
  • / education & outreach /
    • meet a DESI member
    • blog
    • planetarium show
    • DESI high
    • interactive visualizations
    • DESI Merch

Sneaking around with DESI data

January 24, 2023 by

Recovering from the Contreras Fire
Benjamin Weaver, NOIRLab
January 24, 2023

DESI collaborators have previously reported on the Contreras Fire and how it initially affected DESI operations along with an update as personnel were able to return to Kitt Peak National Observatory (KPNO). Even as we were cleaning up KPNO and the DESI instrument in August 2022, we knew that it would possibly be several months before the Tohono O’odham Utility Authority (TOUA) could replace the infrastructure bringing electrical power and line internet (a fiber-optic link) to KPNO. In fact, line internet was not restored until December 7, 2022, almost exactly six months after the start of the fire.

The story of restoring electrical power and the generators used while restoration was in progress is definitely worth telling, and far more lengthy and detailed than the story I’d like to tell now, recovering from the missing internet connection.

The DESI Data Management team, working closely with the on-site NOIRLab personnel, developed a plan to work around the missing internet, even as DESI returned to full-night operations. It wasn’t a new plan; in fact, it was a very old plan: “Sneakernet”. The idea is that a human being (presumably wearing sneakers) can hand-carry data on physical media from one location to another, and in certain circumstances, the effective bandwidth of that transfer can be much, much higher than the bandwidth of an internet connection or other communication channel. This is actually even more true now than it was in the past, because the storage density of physical media has experienced exponential growth similar to that of Moore’s law for CPU performance. However, while Moore’s law has been showing signs of reaching fundamental physical limits, storage density trends continue to rise.

Old school data transfer station at NOIRLab’s Tucson facility

With that in mind, we purchased six 2 TB SanDisk solid-state drives (SSDs), and a few fast USB cables. There is nothing exotic about the hardware; these items would be available in many consumer electronics or department stores. Then with the 6 disks initially carried to KPNO, we would place an entire night of data (or sometimes several nights) on a SSD. One of the personnel working at KPNO would be designated to carry the SSD to Tucson at the end of the work day. Then we would pick up the disk either in person or from a drop box and copy the data to NERSC where it could be processed with the usual DESI data pipeline. Since we purchased six SSDs, that meant we always had several SSDs on standby at KPNO in case an already-transferred SSD could not be brought back from Tucson to KPNO the next day. On weekends, no one would be available to carry the SSDs, so we simply had several nights on Monday evening’s SSD.  From the point of view of the pipeline, the only difference was that an entire night (or several nights) would arrive all at once, after a delay of 1 to 3 days, instead of the normal operations procedure of one exposure at a time, almost immediately after the exposure is completed.

Transfer disk plugged in

The DESI instrument resumed taking data starting with night 20220825 (August 25, 2022). The first SSD was brought to Tucson on September 2, 2022. This process continued until line internet was restored on December 7, 2022. The original data transfer system was ready to go at that point and was reactivated almost immediately after line internet was restored. Thus the last night of data transferred by sneakernet was 20221206 (December 6, 2022). In total, 16,726 exposures over 100(!) nights were hand-carried to Tucson. This amounts to 6.1 TB of data.

Tucson to NERSC data transfer submitted successfully

The bandwidth of hand-carried SSDs is impressive, but so is the bandwidth of modern USB cables. After some experimentation, we discovered that we could transfer data directly from the external SSD to NERSC via Globus Online, with bandwidth performance indistinguishable from the case where the data were first copied to a workstation, then transferred via Globus. In other words, the limiting bandwidth was the internet itself between Tucson and NERSC, not the internal systems of the receiving workstation.

While we were transferring DESI data with a very old system (but with the latest SSDs), we were also experimenting with a new one: Starlink. In fact there were at least two separate Starlink installations active at KPNO during the recovery.  One, which we’ll call the “DESI Starlink”, was a strictly limited command and control channel for use by instrument experts. The DESI Starlink was not used for any significant data transfer. The other, which we’ll call the “NOIRLab Starlink” was used for more general access to all of KPNO. We did use the NOIRLab Starlink for some data transfer, with mixed results.

At least currently, Starlink works best with a point-to-point style networking connection, optimized for individuals or residences, much like a residential cable modem. Since the uplink bandwidth is currently a factor of 10-20 times slower than the downlink bandwidth, it is less practically suited as a two-way connection between two substantially-sized networks, such as between KPNO and the Tucson base facilities. This manifested as a significant, recurring routing problem. Although bandwidth was adequate for interactive use and for the transfer of small data files, the connection was frequently not working and the routing systems had to be monitored constantly. Nevertheless, we were able to use the NOIRLab Starlink to maintain an off-site copy of the DESI operations database. This copy would have been impractical to maintain using daily SSD transfers, because the copy is a real-time mirror of a live database; it is more of a stream than a set of files. There were occasions where the database mirror fell behind due to interruptions in the Starlink connection. In these cases files created by the database were transferred manually on the SSDs and then once in Tucson the data files were manually added to the database mirror.

It was interesting and a good experience to transfer data over sneakernet. It was also a lot of work and hours. I want to give credit to a big team who all contributed to this effort: Keith Blaine, Paul Demmer, Matthew Evatt, Steve Grandi, Bob Marshall, Rod Rutland, David Sprayberry, Christopher Stone, Bob Stupak (NOIRLab), Martin Landriau (LBNL), and Klaus Honscheid (OSU). Here’s one job we hope we never have to do again!

Filed Under: blog, feature on homepage

Primary Sidebar

  • DESI High, the School of the Dark Universe, is Back!
  • DESI’s Evolving Dark Energy Lights up the News
  • At the Big Reveal: DESI’s December 2024 Unblinding Results
  • DESI DR2 Results: March 19 Guide
  • Closing Our Eyes to Truly See — “Blinding” in DESI’s Analysis of Its Cosmological Measurements
  • Charting the Dark Cosmic Web: Where DESI and Imaging Surveys Intersect
  • BaoBan Greets Spectators at 2025 Tohono O’odham Rodeo Parade
  • DESI Finds Black Holes are Common in Small Galaxies Too
  • Brushing Away the Dust to Uncover Cosmology: Examining the Sigma-8 Tension with DESI Galaxies
  • DESI’s 50M Milestone and a “Biggest Breakthrough” Honor
  • DESI 2024 Results: November 19 Guide
  • All is Not Lost: Tiny Groups of Galaxies Remember Their Origins
  • “5000 Eyes” Premieres in Mexico City
  • DESI 2024 Supporting Papers: June 11 Guide
  • DESI Joins in Hosting Kitt Peak Open Night for Tohono O’odham Nation; BaoBan Makes Guest Appearances
  • DESI 2024 in the News: Science Communicators Discuss the Cosmology Results from DESI’s Inaugural Year
  • DESI 2024 in the News: Is Dark Energy Weakening? New Uncertainty Invites Optimism About the Fate of the Universe
  • DESI 2024 Supporting Papers: April 11 Guide
  • DESI 2024 Results: April 4 Guide
  • A Record-Breaking Night
  • BaoBan Spotted at the Tohono O’odham Rodeo
  • A very nearby Type II Supernova in the Galaxy Messier 101
  • The DESI Early Data Release is now available
  • The new DESI ambassador: BaoBan
  • 5000 Eyes: creating the DESI planetarium film
  • Dusty views of the southwestern sky
  • Sneaking around with DESI data
  • Lightning and a lunar eclipse over Kitt Peak
  • Untangling the cosmic web
  • Recovery effort update two months after the Contreras Fire
  • First batch of Year 1 Key Project 1 papers
  • Contreras Fire threatens DESI and Kitt Peak National Observatory
  • One year and 12.8 million galaxy redshifts
  • DESI on a T-shirt (and stickers of course!)
  • Selecting targets for the DESI survey
  • The old is new again: social distancing while mapping the universe
  • Congratulations to Frank Valdes
  • The DESI peculiar velocity survey
  • DESI breaking records
  • A DIY guide for upgrading your $100 million world-class astronomical instrument
  • An upgraded DESI returns to the sky
  • Cosmic cartography
  • Diversity of DESI SV quasars
  • The beginnings of the 3-dimensional map
  • What do DESI’s 5000 eyes see?
  • Plugging away
  • DESI begins search for elusive dark energy
  • DESI begins its “One-Percent Survey”
  • DESI embarks on Survey Validation
  • An undergrad perspective on DESI
  • Women in DESI on International Women’s Day
  • Hunting the oxygen doublet in distant galaxies
  • Congratulations to David Weinberg
  • DESI imaging leaves a legacy at infrared wavelengths
  • DESI target selection
  • DESI successfully completes commissioning phase

Footer

TEAM LOGIN

twitter   instagram   facebook

Copyright © 2018 Dark Energy Spectroscopic Instrument [DESI]

Copyright © 2025 · Parallax Pro DESI on Genesis Framework · WordPress · Log in