You get a Sherpa release, you get a Sherpa release, and you get a Sherpa release

It turns out I haven't been keeping this blog up-to-date with Sherpa releases, so we now have had, since the last post, the following releases:

1. Sherpa 4.12.1, July 14 2020, which was an "emergency" fix to Sherpa 4.12.0 as well as introducing Python 3.8 support;
2. Sherpa 4.12.2, October 27 2020, which focused on plotting and modelling fixes, although there were a number of fixes and improvements related to flux calculations;
3. Sherpa 4.13.0, January 8 2021, which is essentially the same as the Sherpa 4.12.2 release;
4. Sherpa 4.13.1, May 18 2021, which mainly added support for newer versions of NumPy and matplotlib, but also had a few other build updates and bug fixes;
5. Sherpa 4.14.0, October 7 2021, which introduced macOS ARM support (at least for those bits of the system which support the ARM architecture, which unfortunately means not all optional extras are supported at this time), Python 3.9, XSPEC 12.12.0, re-worked filtering and binning code to address a number of edge cases, particularly with PHA data, and improvements to the documentation;
6. Sherpa 4.14.1, May 20 2022, which added support for Python 3.10 and XSPEC 12.12.1, reworks to the plotting and I/O backends (with more to come, but unfortunately not in Sherpa 4.15.0), and a number of internal changes;
7. and Sherpa 4.15.0, October 11 2022, which updates the aging build infrastructure to better match the current Python ecosystem (but as this ecosystem hasn't quite settled down yet, particularly for Python packages like Sherpa which have a bunch of extension modules that require NumPy, means it is an ongoing process), updates to the behavior of the notice and ignore set of commands in the UI layer, changes to how data creation and validation happens to better catch corner cases, and preparation for plotting updates (which unfortunately did not make it this release).

Thanks to the Zenodo site which makes it easier to create a list like this! It also means that as I can link directly to the Sherpa 4.15.0 release notes I'm not going to bother going through the individual changes in this release.

As with previous releases, the Sherpa 4.15.0 release is available via our Conda channel [1], pip [2], Zenodo [3], and GitHub [4]. We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, questions, cookies, and any other positive interaction. Please join in the fun on GitHub: https://github.com/sherpa/sherpa
The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.15.0/

Happy fitting and modelling!

• [1] https://conda.anaconda.org/sherpa/
• [2] https://pypi.python.org/pypi/sherpa/4.15.0
• [3] https://zenodo.org/record/7186379
• [4] https://github.com/sherpa/sherpa

Sherpa 4.12.0 release

Sherpa 4.12.0 has just been released (January 30 2020) and is available via our Conda channel (https://conda.anaconda.org/sherpa/), pip (https://pypi.python.org/pypi/sherpa/4.12.0), Zenodo (https://zenodo.org/record/3631574 , and GitHub (https://github.com/sherpa/sherpa/tree/4.12.0).

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, questions, cookies, and any other positive interaction. Please join in the fun on GitHub: https://github.com/sherpa/sherpa

The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.12.0/

OVERVIEW

This release of Sherpa is based on the CIAO 4.12 release and includes additional
bug fixes and improvements.

This version of Sherpa has been tested with Python 3.5, 3.6, and 3.7. It does not work with Python 3.8 (please follow issue #730 for progress on this). 

This release also provides support for XSPEC 12.10.1 (patch 'a' or later) in
addition to versions 12.10.0, and 12.9.1.

Details

#388 - Address indexing DeprecationWarning/IndexError from NumPy for PHA data with filter + ignore_bad (fix #361)

#602 - Reduce integration tol

Changed integration routine tolerance from double epsilon to float epsilon
since opt routines hove tolerance of ~ 1.0e-7

#606 - stop using numpy.typeNA which is now deprecated

A numpy deprecation warning was fixed by removing the usage of typeNA, which
was not documented and will be removed in a future release of numpy. The
print_fields function has been changed to include a default mapping using
the current typeNA implementation.

#608 - update ciao default plotter to matplotlib

The default Sherpa plotting package has been changed to matplotlib for
CIAO users as well (it had been the default for standalone users for
several years). The update will be applied to new users when the
sherpa command is run. Users of previous versions would need to
edit/regenerate the file.

#616 - enable run-time-option to test cache (default=True)

Capability to allow the user to turn off caching at runtime for testing
purpose.

#622 - Do not create warnings about error bars in plots if no error bars are to be shown (fix #621)

Remove the "The displayed errorbars have been supplied with the data or
calculated using chi2xspecvar..." warnings that appear for data,
residual/ratio, and fit plots when the user has explicitly turned off
the display of errorbars (by setting the yerrorbars plot-preference
setting to False).

#626 - add plot_fit_ratio and plot_bkg_fit_ratio functions

Add the plot_fit_ratio function to the sherpa.ui layer (to match
plot_fit_resid and plot_fit_delchi) and plot_bkg_fit_ratio to the
sherpa.astro.ui layer. Added tests for a number of plot types.

#640 - fix 'tuple index out of range' error in the fit.est_errors method

Sherpa's Confidence class is confused if a parameter is frozen then
thawed after a fit resulting with an exception of the form: IndexError:
tuple index out of range. The error does not show up if conf is called
from the ui level

#647 - fix numpy hist warnings

When plotting a PDF Sherpa now calls numpy.histogram with density=True |
False rather than normed=True|False. There is no change visible to the
user other than a warning that will not be issued anymore. normed=True
was broken for non uniform bins, but our code always produces uniform
bins, so we should never hit the problematic case.

#658 - Handle files with MJD_OBS or MJD-OBS keywords (datastack module)

The sherpa.astro.datastack module has been updated to deal with files
that use the MJD-OBS keyword instead of MJD_OBS. This is only used by
the show_stack routine, which prints out several keywords from each
file in the stack.

#665 - Remove python 2.7 support (fix #498)

Remove support for Python 2.7 from Sherpa. The metadata used by pip now
requires python 3.5 or higher (but not version 4),

#667 - improve matplotlib plot_fit plots (order of data and model)

Changes to Zorder of plot objects drawn by matplotlib to be more readible

#674 - Add link/unlink parameter tests

This PR adds three tests for sherpa's un/link commands.

#677 - XSPEC: build against 12.10.1m by removing support for models

Remove support for XSrelline, XSrelline_lp, and XSrelline_lp_ext models.
These were added in 12.10.1 but removed in 12.10.1m. It does not seem
worth supporting just those versions where they were available.

#680 - Remove the unused parameter in sherpa.astro.optical.LogEmission model (fix #219)

Remove the unused parameter in the sherpa.astro.optical.LogEmission
model. The LogEmission model only has four parameters (fwhm, pos, flux
and skew) consistent within the calc method. The hidden parameter limit
and it doc were removed.

#682 - jointplot now respects the ratio argument (top plot is larger) with matplotlib backend

The matplotlib back end now makes the main (top) plot taller than the
secondary (bottom) plot when using the sherpa.ui.plot_fit_resid and
sherpa.ui.plot_fit_delchi routines (this also holds for the
sherpa.astro.ui variants).

#683 - Remove chips from the documentation

Remove mention of ChIPS from the documentation (both on read the docs
and in the sherpa.astro.ui and sherpa.ui modules), replacing with
matplotlib where appropriate.

#684 - Add default config options

Whenever an option is read from the configuration file (~/.sherpa.rc)
provide a default option in case the configuration file is missing
(or is missing this option).

#685 - raise ModelErr when model using Data1DInt have overlapping bins

Adds a check to regrid make sure the integrated bins do not overlap.
The PR is a fix for issue #569

#688 - Improve documentation and testing for get_source_plot when sent lo/hi arguments

Improve the documentation for the sherpa.astro.ui.get_source_plot routine,
describing how touse the result when the lo or hi arguments are sent.

#691 - add warning when chips is selected but cannot be imported (rebase of #648)

Sherpa now warns the user if Chips is selected as a backend but it is
not available in the installation

#692 - Allow users to override plot preferences when creating a plot

Change to allow kwargs to be specified to change plot preferences at
creation time

#698 - Add warnings in documentation about masked arrays

Updates docstring to indicate that sherpa doesn't support numpy masked
arrays

#701 - a fix for /data/lenin2/Test/CIAO4.11/SherpaRegressionBeta1/45

Enables the testers to turn on (or off) caching at runtime to test #444

#704 - change calc_ftest from delta dof to dof (similar to XSPEC) - fix for #641

Update to make sherpa's calc_ftest provide same results as XSPEC's
ftest (fix to #641)

#705 - modelCacher1d needs to have a couple of deep instead of shallow copy()

Update to make deep copy of cached function values (fix to #673)

#710 - Add warning about masked array to sherpa.astro.ui.load_arrays

Updates docstring to contain warning consistent with #698

#712 Minor typo in docs

Corrects a typo "is is" in the documentation.

#716 Fix for plot_cdf: plot() got an unexpected keyword argument 'clearwindpw'

Recent changes broke plot_cdf() due to a typo 'clearwindpw'. This update
corrects the typo.

#721 Docs: update installation notes for conda environment

Add a warning about the need to set PYTHON_LDFLAGS to ' ' on macOS when
building within an anaconda environment. This is a documentation-only
change.

#725 convert a list to np.array to avoid warning messages

Fixes issue #723 (a lot of warnings while running 'resample_data()' in
CIAO4.12 Sherpa)

#728 a fix cause using load_multi_arfs causes caching error

Fixes issue #717 (unable to fit if script uses load_multi_arfs/rmfs)

Sherpa 4.11.1 release (not-so-belated version)

Sherpa 4.11.1 has just been released (August 1 2019) and is available via our Conda channel (https://conda.anaconda.org/sherpa/), pip (https://pypi.python.org/pypi/sherpa/4.11.1), Zenodo (https://zenodo.org/record/3358134), and GitHub (https://github.com/sherpa/sherpa/tree/4.11.1).

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, questions, cookies, and any other positive interaction. Please join in the fun on GitHub: https://github.com/sherpa/sherpa

The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.11.1/

OVERVIEW

This release of Sherpa introduces several functional improvements and bug fixes,
in particular Sherpa now has support for:

• asymmetric error bars
• PSFs with better pixel resolution than the data
• running optimization in parallel

This version of Sherpa has been tested with Python 2.7, 3.5, 3.6, and 3.7. Support for Python 2.7 is deprecated and will be dropped in a future release. Please upgrade to Python 3.5 or later!

Details

Documentation and infrastructure fixes are not shown.

#630 Fix "get_int_proj does not work when recalc=True" (#543)

get_int_proj did not work when recalc=True on Python 3. This has now been fixed.

#615 Asymmetric Errors

Sherpa now supports asymmetric error bars. Errors can be read through a new

load_ascii_with_errors high level function, or through the new
Data1DAsymmetricErrs class. Sherpa uses bootstrap for estimating the uncertainties.

#585 plot pvalue

Updates to utilize the appropriate response files (ARF and RMF) for X-ray spectra

and changes to the p_value output to 1/(number of simulations) when p_value is 0
and the number of simulations in not large enough.

#596 Run optimization algorithms over multiple cores

This PR enables the user to run the optimization algorithms (DifEvo, LevMar,
and NelderMead) on multi-cores.

#607 PSF rebinning (fix #43)

Sherpa now supports using a PSF with a finer resolution than 2D images. If Sherpa
detects that the PSF has a smaller pixel size than the data, it will evaluate the
model on a "PSF Space" that has the same resolution as the PSF and the same footprint as
the data, then rebin the evaluated model back to the data space for calculating the statistic.

#614 Refactor data classes (fix #563, #627, #628)

Sherpa's basic data classes have been refactored and cleaned up to help
facilitating fixing bugs and implementing new features. New tests were added to reduce
the chances of introducing regressions in the future.

#612 Fix #609 User Model: unable to change parameter values

A regression introduced in Sherpa 4.10.0 presented users from change
user-model parameter values through direct access. This issue has been
fixed. Several tests were added to reduce the chance of regressions in
the future.

Fix #514 - the command line tool sherpa_smoke in the conda package now correctly honors
command line arguments.

Sherpa 4.11.0 release (rather belated)

Sherpa 4.11.0 has been release for quite some time now, what with a release date of 20 February 2019, and I obviously only just noticed as I came to write the 4.11.1 release blog. Oops. Anyway,on with the show: Sherpa is available via our Conda channel (https://conda.anaconda.org/sherpa/), pip (https://pypi.python.org/pypi/sherpa/4.11.0), Zenodo (https://zenodo.org/record/2573885), and GitHub (https://github.com/sherpa/sherpa/tree/4.11.0).

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, questions, cookies, and any other positive interaction. Please join in the fun on GitHub: https://github.com/sherpa/sherpa

The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.11.0/

OVERVIEW

Optimization routines and statistic methods were made more robust with several bug fixes and improvements.

This version of Sherpa has been tested with Python 2.7, 3.5, 3.6, and 3.7. Support for Python 2.7 is deprecated and will be dropped in a future release.

This release also provides support for XSPEC 12.10.1 (patch 'a' or later) in addition to versions 12.10.0 (included in CIAO 4.11.0) and version 12.9.1.

Details

Documentation and infrastructure fixes are not shown.

#444 Improve caching

When fitting multiple datasets simultaneously Sherpa now tries to cache
model evaluations to improve performance.

#465 Support XSPEC 12.10.0 (fix #436)

XSPEC 12.10.0e is now supported

#508 Ensure fields are initialized before use (pileup code)

Resolved an issue where some fields in the C++ extension could have been
uninitialized in a corner case.

#523 Ensure the ui modules have a unique list of exported symbols (fix #502)

Remove the sherpa.utils.erf symbol from the two ui modules as it is
"over-written" by the ModelWrapper-created symbol when applied to
sherpa.models.basic.Erf.

#524 Be explicit about 'invalid escape characters' in strings

Convert strings that contain "invalid" escape characters, which newer
versions of Python complain about, to raw strings by preceding them with a 'r'
character.

#525 Fixed based line radpro_dm test and changed double --> real (template)

#530 Ensure XSPEC 12.10.0 uses ATOMDB 3.0.9 by default

Ensure that the default AtomDB version is 3.0.9 when using XSPEC 12.10.0.
This is to fix an issue with the models-only XSPEC installation in XSPEC
12.10.0 which uses a default version of 3.0.7 but only provides data files for
3.0.9. Without this change models such as XSapec would return all 0's (unless
the AtomDB version was set manually by the user).

#534 XSPEC 12.10.1 support and mtable fix

Sherpa now supports XSPEC 12.10.1. Several issues were fixed to ensure
compatibility with this release. Note that there are known issues with version
12.10.1 (no patches), so at least v12.10.1a must be used.

#537 / #552 sherpa.rc changes

Sherpa does not read the verbosity.level setting from sherpa.rc anymore but
the option is still in the file for backward compatibility

#539 Fix XSPEC models: mtable table models and ismabs (fix #535, #538, #540)

Fix to address issues with XSpec- #535 XSPEC multiplicative table models
(mtable models loaded with load_xstable_model); #538 incorrect evaluation of
the ismabs model; #540 the kerrd model evaluated to 0 for XSPEC 12.10.0 and
later.

#546 Small code cleanup in LevMar

small code cleanup in the LevMar class to match the other classes in that
file.

#564 / #567 Harmonise the handling of dof is zero or negative in fit and calc_stat_info (fix #565)

The fit and calc_stat_info methods now use the same code to calculate
the reduced statistic (rstat) and "quality of fit" (qval) values. This
avoids a TypeError when the number of degrees of freedom is negative in
calc_stat_info and ensures that NaN is returned for both values when the
goodness-of-fit calculation has failed (e.g. the reduced statistic is zero or
less or the statistic is negative), whereas in previous versions the qval
value could be None or 1 depending on the code path.

#576 Fix regression with responses get_x method

Version 4.10.1 introduced a regression which made the get_x() method fail
when called on RMF and ARF objects. This has been resolved.

#583 Update pager code (fix #445 #561)

Replace the use of an external pager (such as more or less) with a
Python one. This means that Sherpa no-longer uses the PAGER environment
variable, and that the screen output from the show_* series of commands
should now appear in the correct location when using a Jupyter notebook or the
spyder application.

Sherpa 4.10.2 bugfix release

Sherpa 4.10.2  has now been released and is available via our Conda channel (https://conda.anaconda.org/sherpa), pip (https://pypi.python.org/pypi/sherpa/4.10.2), Zenodo (https://zenodo.org/record/2275738#.XBPp81WYUeM), and GitHub (https://github.com/sherpa/sherpa/tree/4.10.2).

This release supports Python versions 2.7, 3.5, 3.6, and 3.7.

We thank everyone who contributed to this release with feature requests, bug reports, testing, code
contributions, or questions.  Please join in the fun on GitHub: https://github.com/sherpa/sherpa/

The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.10.2/

OVERVIEW

This release fixes a regression introduced in Sherpa 4.10.1 that was noted by the GammaPy team. As part of the long term PSF rebinning improvements, Sherpa 4.10.1 introduced a check to validate that the data pixel size and the PSF pixel size match. If they don't match, then a warning is issued. The change did not account for an edge error case. Sherpa 4.10.2 solves this problem.

Details

#551 Fix gammapy/gammapy#1905, catch errors with psf pixel size.

Sherpa did not properly catch errors when checking the PSF pixel size matches the image pixel size. This has now been fixed.

Caveats

There is a potential issue for macOS conda users with python > 3.6 and matplotlib v3.0.1, when pyqt5 is not installed and selected as a matplotlib backend. With this specific combination of platform and dependencies, matplotlib can fail to work properly (rhe issue can be replicated without importing Sherpa, so this is not a Sherpa bug).

Several workarounds exist, the simplest of which are:

• install pyqt5 (conda install pyqt`).
• downgrade matplotlib to v2 or v3.0.0 (conda install matplotlib=3.0.0)

Sherpa 4.10.1 release

Sherpa 4.10.1  has now been released and is available via our Conda channel (https://conda.anaconda.org/sherpa), pip (https://pypi.python.org/pypi/sherpa/4.10.1), Zenodo (https://zenodo.org/record/1463962#.W8jkmCSYUeM), and GitHub (https://github.com/sherpa/sherpa/tree/4.10.1).

This release supports Python versions 2.7, 3.5, 3.6, and 3.7.

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, or questions.  Please join in the fun on GitHub: https://github.com/sherpa/sherpa/

Overview

This release fixes several bugs and introduces a few new features, notably the ability to evaluate model components on arbitrary grids and generate user-defined ARFs and RMFs. Also, as of this release Sherpa will no longer rely on any Fortran code. See the following section for details.

It is now possible to build the Sherpa documentation using Sphinx. Additionally, the Sphinx documentation is automatically built and hosted on ReadTheDocs. The version for 4.10.1 can be found at https://sherpa.readthedocs.io/en/4.10.1/

Details

#407  Ensure 0-length array is an error in filter_resp (fix #405)

Add an explicit check in the C++ filter_resp code to error out if the noticed channels array is empty.

#422 Improve error message with wrong xspec version

Improve the handling of XSPEC versions mismatch.

#466 Fix bounding box out-of-bounds memory read

Avoid an out-of-bounds memory read when calling pad_bounding_box (when the data is not matching the expected conditions for this call).

#469 Evaluate model on finer grid

Sherpa users can now define arbitrary grids, called evaluation spaces, on which to evaluate individual model components, both in 1D and 2D. This can be useful in a number of cases, for instance when it is desirable to evaluate models on a finer grid than the one defined by the data, or in convolution models where information outside of the data range can be used to reduce boundary effects or to inform the evaluation inside the data space. 

Also, when plotting individual 1D model components (plot_source_component), if a specific evaluation space was attached to the model then the plot will be of the model evaluated over that evaluation space, not the data space. Other plotting commands should be unaffected by this change.

#470 Order for np.ones() should be a one length string not a bool

Fixed a DeprecationWarning in the optimization module.

#471 Using overwrite rather than clobber of astropy.io.fits

The clobber option in astropy has been deprecated for a while now in favour of overwrite and thus issues a DeprecationWarning. Sherpa now uses overwrite instead.

#475 Fix unecessary runtime warning (fix #402)

A condition check in the optimization code was modified so as not to produce a warning when the value is a NaN.

#481 Remove Fortran code

Sherpa does not rely on any Fortran routines anymore, except those compiled in XSPEC. The existing Fortran code, which was used mainly in some of the optimization routines, has  been replaced by equivalent C++ code. This means that gfortran is no longer required for building Sherpa, although a version of libgfortran compatible with the one used to link the XSPEC libraries would still be needed at runtime if the XSPEC extension is built.

#482 ARF/RMF creation functions

The create_rmf and create_arf functions now allow users to easily generate user defined response objects.

#486 ZeroDivisionError in calc_stat_info (fix #476)

Sherpa did not catch divisions by zero in the calc_stat_info command. That has been fixed.

#484 Equivalent Width errors

Several optional arguments (params=None, error=False, otherids=(), niter=1000), were added to eqwidth. If error = True, then get_draws shall be run if the fit stat is one of the following: (Cash,  CStat, WStat) otherwise a multivariate normal distribution shall be run to generate the samples. The optional niter parameter is used to generate the number of samples. The optional parameter otherids is only used if get_draws is used internally when multiple data sets were used to fit. Alternatively, the user can enter the samples via the params option where the samples must be a numpy.ndarray of dimension (num, npar).

#487 PSF bin size warning

Sherpa assumes that the PSF image and the data have the same pixel size. When this is not true Sherpa ignores the difference, which results in a larger PSF being applied. From now on Sherpa will issue a warning when the PSF image pixel size and the data image pixel size are different.