Claire Poppett, Lawrence Berkeley National Laboratory
October 1, 2021
Anybody who has observed with the DESI instrument has experienced uncommunicable positioners on the focal plane that lead to frustrating pauses in the observing schedule whilst the lead observer or a focal plane expert work to bring these devices back. During the Tuscon summer monsoon season in 2021 a team of DESI collaboration members worked with NOIRLab staff to upgrade the DESI instrument and make these experiences a problem of the past.
To understand why these positioners become uncommunicable, we must first understand how they communicate.
The DESI positioners communicate via a Controller Area Network (CAN bus) and are grouped into patches of 25-75 positioners per CAN bus. Unfortunately, we have discovered that if a single positioner has a faulty communication signal it can affect the other positioners in that patch. The bigger issue here is that this loss in communications leads to a loss in their telemetry and the possibility that they may exceed our temperature limits without our knowledge. The collaboration agreed that this problem needed to be fixed and so a remediation plan was developed.
It was determined that the simplest way to be able to turn off a faulty positioner was to add a relay switch into the hardware but this required a major upgrade to the architecture of the focal plane. This was not possible without removing the focal plane from the telescope and so at the end of July, NOIRLab staff began the arduous task of getting the telescope into a state that made it possible to remove focal plane petals and install them onto work carts staged in temporary clean tents on the C-floor of the Mayall.
Once the petals were on the floor every single fiber and every single positioner wire had to be handled in order to remove the old communications boards from the petals in order to allow new boards with relay switches to be installed. These upgrades were performed on all 10 petals and included extensive testing to ensure that all devices now communicated as reliably as possible. The final petal was inserted back into the focal plane on August 25th and by lunchtime the next day we were able to declare that we had a healthy focal plane.
After only 10 weeks of downtime, the instrument was returned to service during the week of September 12th. Recalibration of the focal plane. Main Survey observations re-commenced on the evening of September 21. Despite the full moon, 60,000 Bright Galaxy Survey redshifts were measured that first night back!
All signs so far point to a truly upgraded DESI focal plane and we can look forward to many more years of DESI science!
