For those interested in failures and recovery in far away spacecraft, check out this thread in August, when Voyager 2 lost contact with earth due to a mispointed antenna (caused by operator error 
).
https://fosstodon.org/@AkaSci/110831401826701180
#Voyager
5/n
Richard Stephenson of DSN Canberra explains on twitter how NASA verified that the uplink is working.
They sent a command to Voyager 1 to switch between non-coherent mode and coherent mode transmission.
In coherent mode, the Transmission clock is derived from the Rx signal instead of from the AUX oscillator. This changes the Tx RF frequency a bit which was detected at the DSN.
https://science.nasa.gov/learn/basics-of-space-flight/chapter10-1/
https://descanso.jpl.nasa.gov/DPSummary/Descanso4--Voyager_new.pdf
#Voyager
6/n
Edited 1y ago
In the blog post at https://blogs.nasa.gov/sunspot/, Voyager engineers point out the difficulty in diagnosing problems and crafting solutions for a spacecraft with a signal round-trip-time of almost 2 days and hardware/software developed over 46 years ago using technology long since obsolete.
"Finding solutions to challenges the probes encounter often entails consulting original, decades-old documents written by engineers who didn’t anticipate the issues that are arising today."
#Voyager
7/n
Edited 1y ago
NASA DSN in Goldstone, CA is currently receiving the downlink from Voyager 1 at a reduced rate of 40 bps. No uplink at this moment.
Apparently, Voyager 1 switched data rate (160 -> 40 bps) & did a full memory read-out of her Attitude and Articulation Control Subsystem, Flight Data Subsystem, and Command Computer Subsystems A&B.
Transmission time = 6 hours
Download size = ~108 kBytes
Here's hoping that the received data is not 0101... 🤞
Edited 1y ago
A similar but not identical problem afflicted Voyager 2 in 2010. Received science data (but not engg data?) was garbled.
The problem was traced to a flipped bit in the program stored in the FDS. A command was sent to flip the bit.
The issue was diagnosed by downloading a full memory image, which implies that engg data download was working.
This is probably what was done today with Voyager 1 today. Hopefully, it is a similar problem.
https://voyager.jpl.nasa.gov/news/details.php?article_id=16
@destevez
#Voyager
9/n
NASA did not provide a date but it looks like this issue was discovered and acted upon on Dec 7 or 8.
The graphic below shows the schedule for Voyager 1 comms via DSN, generated on Dec 7. Normally, the downlink rate is 160 bps. On Dec 8, it was switched to 40 bps. And again on Dec 10. Some special commands for the FDS were also sent.
Since then, the D/L rate has been switched between 160 bps and 40 bps a few times with additional FDS commands uploaded.
https://voyager.jpl.nasa.gov/pdf/sfos2023pdf/23_12_07-23_12_25.sfos.pdf
#Voyager
10/n
Edited 1y ago
Two-way comms happening now between Voyager 1 and NASA DSN Canberra.
Of course, the results of the uplink commands will arrive 45 hours from now. The data arriving now left Voyager 1 22.5 hours ago.
Downlink rate is the lower 40 bps rate.
The DSN schedule for Voyager 1 shown below was modified and published yesterday.
Here's hoping that Voyager engineers are getting closer to a solution 🤞
https://eyes.nasa.gov/dsn/dsn.html
https://voyager.jpl.nasa.gov/pdf/sfos2023pdf/23_12_14-24_01_01.sfos.pdf
#Voyager
11/n
Edited 1y ago
NASA JPL provided a minor update today about the status of the Voyager 1 spacecraft, indicating that the comm. problem that started more than 2 months ago has not been resolved yet. No other details.
Please check out the rest of this thread for more info on the problem where instead of sending science and engg. data, Voyager 1 has been stuck sending a 0101 bit pattern.
@NSFVoyager2
#Voyager
12/n
Edited 1y ago
No new info on the status of the Voyager 1 spacecraft, which since Sep 2023 has been sending a 1010 bit pattern instead of real data.
Several popular science outfits have been covering it lately. A bit flip in the FDS is suspected, but it is difficult to identify since the memory cannot be read back.
Several commands were sent yesterday to Voyager 1; responses will arrive 45 hours later tomorrow.
Wonder why they cannot overwrite all prog and data memory.
https://arstechnica.com/space/2024/02/humanitys-most-distant-space-probe-jeopardized-by-computer-glitch/
#Space
13/n
Good news from the Voyager 1 spacecraft that has been stuck sending a 0101 pattern since Nov 2023.
The team has long suspected the root cause to be a corrupted area of memory in the FDS computer. On Mar 1, they sent some commands to make the FDS skip around sections of memory. The data stream rcvd 45 hours later looked different and was decoded to contain a read-out of the entire FDS memory!
Hopefully, they can now identify and fix the offending memory words.
🤞
https://blogs.nasa.gov/sunspot/
14/n
Voyager is not out of the woods yet, but the lesson for all of us is to never ever give up.
Here is the schedule for comms with Voyager 1 via NASA DSN this weekend. Some new commands will be sent on Friday, with responses expected 45 hours later on Sunday.
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_03_14-24_04_01.sfos.pdf
15/n
Some tech. info on the Voyager FDS computer –
- There was a backup FDS unit but it failed in 1981.
- Custom CMOS CPU - 36 instructions. 80 KIPS, 115 kbps data rate.
- 128 registers, kept in memory.
- CMOS memory, a first in space, 8KB.
- No separate memory for program storage vs execution. The CMOS memory is non-volatile kept powered on by the RTG.
- DMA access to memory by hardware. Instead of “cycle-stealing”, the instructions indicated cycles where DMA can occur.
https://ntrs.nasa.gov/api/citations/19880069935/downloads/19880069935_Optimized.pdf
16/n
Edited 1y ago
Status update on the Voyager 1 spacecraft which has been sending a 0101 pattern since Nov 2023.
The problem seems to be a failed memory part in the FDS computer; engineers are planning to move ~200 words of software from one region to another, according to Joseph Westlake, director of NASA’s heliophysics division, who was speaking at a March 20 meeting of the National Academies’ Committee on Solar and Space Physics.
Westlake sounded very optimistic.
👍
https://www.nationalacademies.org/documents/embed/link/LF2255DA3DD1C41C0A42D3BEF0989ACAECE3053A6A9B/file/D727AF88E8C806D7A1F75C8401AF9CF23BCCC2EC9F3A?noSaveAs=1
17/n
It looks like the Voyager team is preparing for a new "memory upload" to the FDS computer on Friday, as evident from the DSN schedule and instructions shown below for Voyager 1.
I am guessing that this is to rearrange the software so that it no longer uses the locations in the faulty memory chip in the FDS. If true, then hopefully we will hear Voyager 1's true voice on Sunday, 45 hours later. OTOH, this may be just one of many steps on the road to recovery.
🤞
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_03_28-24_04_15.sfos.pdf
18/n
Edited 1y ago
Looks like the "memory upload" to the Flight Data Subsystem (FDS) on Voyager 1 is taking place at this time from the NASA DSN site in Canberra.
Go Voyager!
https://eyes.nasa.gov/dsn/dsn.html
https://en.wikipedia.org/wiki/Canberra_Deep_Space_Communication_Complex
19/n
Edited 1y ago
It's been 6 hours since the "memory upload" data was transmitted to Voyager 1 from the NASA DSN site in Canberra.
During that time, the signal has traveled about a quarter of the way to Voyager 1, about the average distance to Pluto. The response will arrive at earth on Sunday around 1500 UTC (RTT = 45 hours).
Let's imagine a spacecraft sent to the nearest star Proxima Centauri, 4.2 light-years away. How would we diagnose problems and upload new software to it?
Edited 1y ago
All eyes and ears on Voyager 1 as data is being downloaded from it in response to the "memory upload" commands and data sent 45 hours ago.
The DSN Goldstone 70m antenna is receiving data now at 40 bits/s.
Hopefully, the problem with its transmission being stuck at 0101 has been fixed. We will find out this week ...
🤞
Edited 1y ago
NASA Voyager twitter site confirming that Friday's memory upload was intended as a fix for the Voyager 1 transmission problem caused by the failed memory chip in the FDS computer.
"Sister @NASAVoyager's reply to Friday's upload should b arriving now @ Goldstone's 70m antenna DSS-14, hopefully confirming that the Flight Data Subsystem memory update was successful. If so, telemetry should now give clearly interpretable signals with science & engineering data!"
22/n
Great news on Voyager 1.
Richard Stephenson of DSN Canberra reports that engineering data was being received from Voyager 1 last night at 40 bps.
No science data yet, perhaps because they did not switch to the higher 160 bps rate, but this is a major step towards recovery and validates the diagnosis (failed memory chip in the FDS computer) and fix (rearrange software to bypass the failed memory area).
Now waiting for a status update from NASA.
23/n
Edited 1y ago
OK nerds: who wants to create a computer game in which you are the engineer trying to debug Voyager with a half-day lightspeed delay on each move? Turn-based (one move every 12 hours); you have a spec for a very (very) simple microprocessor and some fault data to start with, and your job is to figure out what's gone wrong and patch it. Sits squarely in the genre of sit-and-think puzzle games and https://tomorrowcorporation.com/humanresourcemachine, and would be a fun alternative to Advent of Code if it's ready for December.
Minor update from the NASA Voyager 1 team -
- Confirmation that the problem is due to a faulty memory chip, which affects about 3% of the FDS memory. We have known about this diagnosis since March 27 but if was not officially announced until today.
- No info on the recent report that engg data was received last Sunday.
- "It may take weeks or months" to fix the problem so that Voyager 1 can operate normally without the unusable memory hardware.
The DSN schedule for Voyager 1 next week shows commands to be sent on Tuesday to relocate some FDS code around "VIM5". Presumably, VIM5 is a memory module.
Additional uploads will take place on Thursday.
So, there is still work that lies ahead to rearrange the code around the failed memory chip. And we suppose, it has to be done incrementally and meticulously so as to not accidentally brick the FDS computer.
https://voyager.jpl.nasa.gov/pdf/sfos2024pdf/24_04_04-24_04_22.sfos.pdf
25/n
Edited 1y ago