Introduction
It’s been a while since the last release, especially right after 1.2.2 where we said we would instead pace things up on releases (especially with most people using nightly builds nowadays), so what happened?
Some SatDump History
To explain what happened, it’s useful to first remember the history of SatDump as a project especially since the userbase has changed a lot since those early times.
Do note that this of course is heavily summarized.
Back around 2019, I (Aang23 F4LAU) wanted to get into HRPT, especially newer satellites such as MetOp (and later FengYun). Since pretty much nothing was available at the time, I ended up attempting to write them myself and thanks to Martin Blaho’s old work & others, ended up publishing an initial MetOp decoder soon followed by a FengYun-3 decoder (https://github.com/altillimity/L-Band-Decoder).
From there, I also slowly added decoders for the instruments (imagers, etc). Zbychu SP5EWS also joined the project at the time by contributing an HIRS decoder. Things only carried on with X-Band decoders (https://github.com/altillimity/X-Band-Decoders) and S-Band decoders (https://github.com/altillimity/S-Band-Decoders) as well as a few more such as GVAR, S-VISSR… All that to say that after a while, those little decoders had grown to a pretty large set of tools.
The first public SatDump version, 0.0.1
Around the same time (and potentially helped by the availability of decoders), the satellite reception hobby grew quite a bit and attracted more and more users wanting something less cumbersome to utilize. Many suggested packaging them together into something nicer - and that’s how the SatDump project really started. The first version was really just all the old tools packages into one thing that simply ran them one after the other with an added GUI.
Over time, more and more features were added such as projections, calibration for some instruments, far more satellites… And it kinda got messy. Working on the codebase around 0.0.39 was painful, and it was clear there had been more than one massive design mistakes made initially in large due to not really grasping what would really be needed for further processing and features at first.
SatDump 0.0.38
That’s why back then Zbychu & I ended up stopping development on master for a while in order to work on what 1.0.0 would become. This involved several months of work to introduce the concept of “products” to store instrument data, the viewer to allow processing them in a much more user-friendly way, a new recorder…
And from then on, that’s what every 1.x.x release has been based on. Of course things changed a lot since, but the core is still similar (and in fact, most code for 1.0.0 could still function in 1.2.2). It’s also worth pointing out that while 1.0.0 introduced pretty important new concepts, it wasn’t a deep rewrite either and downsides of legacy 0.0.39 codes still prevails in some situations.
Therefore, as you might have guessed already… There we go again! :-) Things in 1.2.3 were really starting to show their limits and actively preventing really going forward. Another (stupid) issue was a plain lack of documentation, leading more than once to breaking something pretty badly entirely by mistake due to a lack of understanding of what code dating back from the first decoders cited here did… Yes, kinda sucks.
So yes, it turns out development never stopped but has rather been focused on another large rewrite meant to finally fix design mistakes and completely overhaul several parts of SatDump (featuring Documentation!!!!!).
What is it about?
This is more meant to be an overview of the changes rather than a complete description, both because it’s near impossible to list them all and because most of it is covered in the documentation already
Product System
At first glance the concept stays identical. It’s still one instrument per product and a concept of dataset to group them together. However, under the hood things were heavily changes and especially simplified.
Alignment / Projections :
First of all, a product can now only store channels with a single projection configuration, shared between all of them. Channels also must be correlated between them by predefined functions (affine transforms or more arbitrary options). That honestly doesn’t change much for the user except the ability to align channels that need more complex transformations (such as a rotation or non-linear transform for the VIIRS DNB channel) to be aligned, and forcing instruments that cannot be correlated such as ASCAT to be split.
Arctic Weather Satellite’s MWR showing a rotation of a channel being performed by SatDump to align it with the others
Calibration :
The old system was rather inflexible, it only allowed 2 hardcoded units : Either albedo, or radiance. Conversions were hardcoded making it near-impossible to really add any other unit and such. Plus, sometimes it was required to calculate other types of variables from the data such as the sun elevation for each pixel.
Hence, the new system made that fully generic. There are still default units for common cases, but channels can be calibrated in any that may be needed. Additionally, converting between units is done via “converters” that can be added as needed. For example, it is now possible to obtain brightness temperatures in Celsius directly.
Those changes also allow using more than one unit from the same channel in the same expression.
Sun compensation was also added. This works in two ways: firstly the raw sun angle can be extracted and used as a “channel” in the expression, or alternatively the reflected radiance for a specific channel can be corrected and output as a calibrated value.
Sun-angle compensated, reflective radiance calibrated imagery from NOAA 15
GUI Changes - No more tabs & Explorer (previously - Viewer) changes
Alongside the product rework, the entire concept of the viewer was reworked as well. Previously, it was limited to only loading datasets and products, and would only allow performing a single projection at a time. That worked in most cases, but it was pretty limited and horribly specific, as all automatic and GUI processing code was also separate.
The idea this time around was to go with something more similar to most GIS programs or file explorers, that is a tree viewer with elements (a dataset?) and sub-elements (products?).
Therefore now the viewer’s base isn’t actually specific at all anymore. All it does is display handlers, and allow dragging them around into one another. That simple concept allows handling anything from a dataset containing several products, to a projection containing several layers and overlays… And far, far more things. It also makes for a nice place to include smaller tools (such as the WIP Binary viewer, the LUT editor) in a less intrusive way. This is in large why it has also been renamed to Explorer, to better reflect the wider role it now plays.
Welcome screen from SatDump 2.0.0
It also turned out that this new approach was so versatile that it could actually replace the entire GUI. Therefore now, everything from running pipelines to recorders are added as “handlers” on the left panel.
This has a few important benefits :
- Adding multiple recorders, or running several pipelines at once is trivial,
- It doesn’t require loading up everything even if the user won’t use it (especially the recorder), which allows SatDump to not only keep a lower footprint but also load a lot faster,
- As nothing is hardcoded in the GUI anymore (outside of essentials), it’ll be a lot easier to maintain (and this has already shown!).
An in-use session of SatDump 2.0.0
Therefore, things moved around a bit in the GUI. Offline processing (now just Processing) is now in the top-level menu, just like the official products loader.
Another major aspect is the introduction of dataset-wide processing via scripts (and potentially flowgraphs as well) to allow doing multi-instrument composites and much more.
Pipeline format
The pipeline format had not changed nearly at all since the first commit of SatDump. It allowed doing a lot more than was ever actually used leading to pretty convoluted configuration. As an example, each pipeline step would allow for running several modules… All that to say, it’s a lot simpler now.
A sample pipeline file, here for the POES constellation
First-Party Products & xRIT files support
Previously, first-party products (that is, products distributed by EUMETSAT, NOAA, etc themselves in their processed forms) had to be loaded in by first processing with a pipeline, which would then save it as products somewhere on the system before loading it again.
However, considering that the primary usecase there is to load them up to look at them in SatDump, that scheme was a bit unecessary and cumbersome. The system was entirely re-done to allow simply dragging or opening them like any other file. Furthermore, this new system does not care about whether the files are inside a zip, .bz2 file or inside a random folder.
Additionally, support for most xRIT formats (LRIT/HRIT/UHRIT files) was added in the same way to allow loading ELEKTRO-L & MSG data transmitted on EUMETCast. It is now possible to simply open those files and SatDump will take care of it.
The same xRIT system is also now common to all xRIT downlinks, making it a lot easier to maintain.
Meteosat-9 IODC HRIT from EUMETCAST
Of course, processing via a pipeline or downloading via the downloader is still possible.
Equation → Expression
Most of the composites that used Lua in legacy SatDump didn’t necessarily need the full power of a scripting system. For example, NDVI used Lua just because the LUT system could only handle a single channel, while NDVI is an index produced from two channels which is then applied to a LUT. Split window enhancements such as water vapor split window were also handled with Lua for the same reason.
This was quite obviously far from ideal, because Lua is slow and with larger and larger image sizes from more advanced instruments (such as FCI) it quickly resulted in a bottleneck.
Therefore, the new Expression Editor does double duty: it can both function as the old Equation Editor with simple RGB enhancements, math operations and ternary operators, but it also supports C-like macros, 1D and 2D LUTs as well as equirectangular projections to use as background or cloud masks.
Additionally, this editor was tightly integrated with the calibration system. It is possible to add a macro representing a calibrated channel, with user defined parameters, using just a few mouse clicks from the GUI.
While this is more limited than a full-blown scripting system, it allows to cover for the majority of the use cases, and it is substantially faster than Lua (and even the old Equation Editor!). Even fairly complex composites such as Sea Surface Temperature can be implemented within the Expression Editor.
When combining channels that have different orientations or resolutions, the ref_channel option can be used to select which one will determine the final composite’s parameters.
Composite rework
Early on, SatDump distinguished itself from other software by offering a wide variety of RGB composite imagery. However, the early approach had some drawbacks, namely that the input images weren’t calibrated at all and that most of the composites were tuned for the normal imagery that the author was receiving. This became a problem when SatDump started growing outside of Western and Eastern Europe, as most composites tended to “break” in unexpected ways when exposed to a different morphology such as when a significant portion of the image is occupied by the Sahara Desert.
Examples of a composite breaking as the land in the image gets progressively more arid
With the introduction of calibrated imagery, the infrared portion got an in-depth overhaul with many useful composites and products such as Cloud Top IR, Night Microphysics and more. However, the daytime composites using mostly visible were not really touched up more than strictly necessary, given that for most people they were not a problem. Mostly, the White Balance and Equalize features were heavily used to correct most of the glaring issues, when they were present.
With sun compensated calibration now available as well as a general push forward to use SatDump in a more professional environement, it was clear this approach was far from ideal, therefore most composites were redone to be calibrated in an absolute fashion and not depend anymore on features such as White Balance whose output is unpredictable with respect to the input.
Composites calibrated against absolute values
Additionally, most of the composites used in operational meteorology were thoroughly checked and validated against primary sources, to ensure they perform adequately. Meteosat (MSG/MTG) composites were added and validated as well, especially those using HRV such as Fog HRV and Cloud HRV.
Planned release “schedule”
By design sure a large rework will make the software absolutely unusable for a little while, therefore all the work had to be performed on another branch in the meantime. However, it can’t last forever and has to be merged back in master at some point.
The way we decided to do it was to keep it separate until the bulk of the rework that would impact the user most & break compatibility the most was done, and from there start releasing alpha releases until the full thing is done to finally release 2.0.0.
⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️
As those alpha 2.0.0 releases are both breaking compatibility and potentially more unstable, you should NOT update from 1.2.3 if you expect things to be reliable - or do testing on your end first. Also make sure to read this article in full to understand the changes that may affect your usecase.
⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️⚠️
Therefore, without any specific timeline here’s what to expect :
- A merge of the rework (branch
verywip) into master, - For at least a few months, relatively frequent alpha releases (as nightly builds),
- Whenever things are done, a proper 2.0.0 release.
Changes
Common enhancements
On most instruments the basic composites, especially for infrared, are identical. This lead to lots of repetitions and inconsistencies, where not all enhancements were available for every instrument even though the instruments themselves were more than capable of outputting these enhancements. With the new common enhancements, all it takes to enable them for a new instrument is a single line of code, streamlining the process and reducing the chance for mistakes.
LUT Generator
To help with the creation of new composites, a LUT Generator was added into SatDump that is similar to the Color Table tool in McIDAS. It is designed to be used to easily create look-up tables, display them on a sample image, and save the result to a file for later use into the Expression Editor. Compared to the one in McIDAS, it also supports a color picker (currently working only on Linux) to easily pick colors (from a website, image, research paper…), undo functionality and a real time preview.
The LUT Generator working on a Suomi NPP image
Crop functionality
It is now possible to crop a region of a satellite image directly into SatDump. Then, the crop can be projected or saved. Additionally, all of the image controls such as white balance, contrast, hue/saturation and more, work also on the crop.
Cropping the gulf of Naples from Meteor-M N°2-4 KMSS
Shapefiles support
Previously, SatDump made use of shapefiles (*.shp) internally to provide country lines, coastlines and cities overlays for maps. This approach was very limited and some users resorted to manually changing the built-in shapefiles with different ones, for example to add county or state lines.
In SatDump 2.0.0 it is now possible to add any shapefile and have it behave as a layer on an image or a projection. Built-in shapefiles from legacy SatDump are also there as before.
A country shapefile in use on NOAA-15
Lua → Angelscript
While the Expression Editor covers almost all cases, there is still the need for a more flexible solution in some use case. Some examples include complex enhancements like Land Surface Temperature that need databases for emissivity, or projecting two different instruments on top of each other for blended composites. Additionally, a scripting solution is useful to interface with third-party software or to do ancillary tasks, such as packaging satellite receptions for archival.
Lua had significant issues with speed and flexibility, so it was decided to switch to Angelscript for these purposes. Angelscript provides a C++ like syntax, is very fast, and integrates seamlessly with the SatDump architecture.
Hue and saturation controls
For certain instruments, such as MSU-GS, it is necessary to shift the hue of the image by some amount to make the image more realistic and lifelike. This was traditionally done directly in C++, but this approach, in addition to being hardcoded and specific to each instrument, clashed with the goal of having every satellite output SatDump products.
Hue and saturation controls were therefore added, borrowing the system used in GIMP, both to the GUI as well as to the Expression Editor. Hues can be shifted and saturation adjusted either on a primary color basis, or for the entire composite image.
VIIRS DNB Channel back into the main VIIRS Product
VIIRS, unlike most instrument has channels that are not downlinked in a linear fashion. This means that normal channels (Imagery / Moderate) could not be merged together with DNB in order to do composites without first re-projecting everything. Therefore, to allow projections it had been split into another product entirely.
Thanks to SatDump’s new composite system now allowing more or less arbitrary transforms between instrument channels, it is now possible to have it back with the rest of the channels and do composites with it. SatDump will automatically correlate the channels together.
Suomi NPP composite akin to HRV Cloud using VIIRS channels I4 and DNB
MSU-MR Visible Calibration
In a previous release, METEOR-M’s MSU-MR instrument had already seen the addition of calibration on its infrared channels. Visible had been left uncalibrated. Fortunately, since it turned out to be relatively simple & stable, calibration could be added based on official data provided by Planeta.
MSU-MR is now fully calibrated.
Initial SSTV Support
Following a large amount of requests, an initial (and still experimental) SSTV decoder was added to SatDump to cover transmissions such as the ISS’ PD120, FRAM2’s Robot36 and so on. Feedback is appreciated to test this further.
Example of decoding SSTV
GK-2A UHRIT Support
Thanks to data provided by John Bell, Drew (@pluggerlockett) and Aybar/Serge, full support for GK-2A was added to SatDump. This includes demodulation and all processing down to calibrated images.
ELEKTRO-L (MSU-GS) LRIT/HRIT Calibration
Until now, ELEKTRO-L HRIT/LRIT was supported but did not allow for calibrating received images. Thanks to more general work on xRIT file processing as described earlier, this is now possible (note : LRIT is not sent with calibration data on ELEKTRO).
Elektro-L N°3 HRIT showing fully calibrated infrared, like all other satellites transmitting HRIT
MSG (SEVIRI) LRIT/HRIT Support
Support for loading & processing SEVIRI LRIT/HRIT files was added. While the downlinks themselves are not active anymore, those files are still broadcast on EUMETCast.
Meteosat-11 SEVIRI from EUMETCAST
FengYun-2 / Fengyun-3
While this is still considered experimental and some features are missing, it is now possible to load FengYun-2 and FengYun-3 HDF5 files in SatDump.
FengYun-3F HDF from CMA’s archives
GVAR Improvements & Products (Projections & Calibration)
Unlike all LRIT/HRIT geostationary satellites, GVAR had been left out and pretty much left untouched for years. Not anymore! :
- The decoder saw improvements to significantly improve reliability at lower SNR,
- It is now saved as products which can be loaded in the viewer, projected and calibrated (assuming the SNR is good enough).
It is therefore now on par with the usual level of support on other satellites.
A calibrated & overlayed composite from SP5EWS’ station
A RAMSDIS Water Vapour Enhancement from bosslike5’s station
FY-2 S-VISSR Improvements & Products
Similarly to GVAR, S-VISSR is now provided as products. Projections and calibration are not currently supported however, and satellites are now identified properly.
Thanks Meti for his work on this!
FengYun-2H S-VISSR received by Meti
FengYun-4 LRIT/HRIT Improvements
While this support is still pretty experimental (mostly due to the poor quality of data disseminated in the clear), this data will now be provided as products just like any other xRIT downlink.
Force X11 on Linux
As more GNU/Linux distributions start defaulting to Wayland, it had become increasingly obvious that GLFW’s Wayland backend was not really in great shape at the moment. Things like Drag & Drop kept crashing, sometimes window decorations wouldn’t show… etc.
Therefore the decision was made to have it automatically default to the X11 backend (even on Wayland). This only results in it running on XWayland instead of Wayland directly, and therefore changes nothing for users.
Better Arctic Weather Satellite MWR support
While AWS support was present already in legacy SatDump versions, channels were clearly not aligned as it was not possible to do the necessary corrections previously. This is now fixed, allowing for nice composites!
KMSS Channel alignment
This is just re-iterating what was explained earlier, but identically to AWS, METEOR KMSS needed more advanced transforms in order to align channels properly.
Currently only implemented on METEOR-M 2-4.
Channel alignment on Meteor-M N°2-4 KMSS
UVSQSat-NG Support
Support for the UVSQSat-NG satellite was added. Currently decoding of camera images is possible, and later on some processing for other instruments may be added.
UVSQSat-NG image
More information in the window title
It was sometime a problem for people to know if they were running SatDump in Release or Debug mode and the branch in use. Therefore, now all this information is simply present in the title of the windows, or name of the version when running in CLI.
Welcome Screen
Following the rest of the UI changes, we decided to take you a trip back in time by adding old-fashioned daily tips and shortcuts to the GUI when SatDump is opened! ;-)
The gorgeous Welcome Screen shown here with the beautiful Graphics Design theme by Júlia
Initial SSMIS Support
The Special Sensor Microwave Imager/Sounder is a microwave sounder on board DMSP satellites. Its high resolution compared to other instruments such as ATMS or MHS make it the preferred tool of meteorologists to analyze and quantify parameters in cyclones and hurricanes. SatDump can now extract SSMIS data from the DMSP RTD direct-broadcast link. Some work is still needed to get projections and potentially calibration running however. This was prompted by the now cancelled plans to discontinue distribution of SSMIS data. The reason why these plans were shelved can be guessed by the reader ;)
Support for files provided on the GPM FTP server was also added (L1C files).
Instrument Config editor (may be temporary)
Aria Horak (@Aweeri) also contributed an instrument configuration editor to make editing composites easier with the new format.
Note : as mentionned above, considering further planned changes this will likely change
Projection changes
With the new GUI and overall system, projections also had to be fully redone. It is now possible to add several projections at once, and add as many layers you want. Otherwise features are mostly identical to what it was before.
Note : Projections will get a further rework later on!
Radiation Products => Punctiform Products
One type of data that wasn’t handled properly at all by previous SatDump versions were single-point products such as Jason-3, telemetry and so on. This has been redone, and while it’s not entirely finished at the moment it is already more versatile than before.
This allowed bringing back Jason-3 processing, all and new radiation instruments and TIP telemetry from POES thanks to a PR by Thomas.
Sample Radiation data from Jason-3’s LPT instrument showing the well-known SAA
Doxygen documentation
All of SatDump’s documentation was switched over to Doxygen. This makes it easier to manage for us, and allows documenting the code better for users wishing to utilise the API.
You can find it at https://docs.satdump.org/v2
Bugs fixed
Too many to even begin trying to fill that section…