Thursday, August 1, 2019

Sherpa 4.11.1 release (not-so-belated version)

Sherpa 4.11.1 has just been released (August 1 2019) and is available via our Conda channel (https://conda.anaconda.org/sherpa/), pip (https://pypi.python.org/pypi/sherpa/4.11.1), Zenodo (https://zenodo.org/record/3358134), and GitHub (https://github.com/sherpa/sherpa/tree/4.11.1).


We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, questions, cookies, and any other positive interaction. Please join in the fun on GitHub: https://github.com/sherpa/sherpa
The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.11.1/

OVERVIEW


This release of Sherpa introduces several functional improvements and bug fixes,
in particular Sherpa now has support for:
  • asymmetric error bars
  • PSFs with better pixel resolution than the data
  • running optimization in parallel
This version of Sherpa has been tested with Python 2.7, 3.5, 3.6, and 3.7. Support for Python 2.7 is deprecated and will be dropped in a future release. Please upgrade to Python 3.5 or later!

Details

Documentation and infrastructure fixes are not shown.
#630 Fix "get_int_proj does not work when recalc=True" (#543)
get_int_proj did not work when recalc=True on Python 3. This has now been fixed.
#615 Asymmetric Errors
Sherpa now supports asymmetric error bars. Errors can be read through a new
load_ascii_with_errors high level function, or through the new
Data1DAsymmetricErrs class. Sherpa uses bootstrap for estimating the uncertainties.
#585 plot pvalue
Updates to utilize the appropriate response files (ARF and RMF) for X-ray spectra
and changes to the p_value output to 1/(number of simulations) when p_value is 0
and the number of simulations in not large enough.
#596 Run optimization algorithms over multiple cores
This PR enables the user to run the optimization algorithms (DifEvo, LevMar,
and NelderMead) on multi-cores.
#607 PSF rebinning (fix #43)
Sherpa now supports using a PSF with a finer resolution than 2D images. If Sherpa
detects that the PSF has a smaller pixel size than the data, it will evaluate the
model on a "PSF Space" that has the same resolution as the PSF and the same footprint as
the data, then rebin the evaluated model back to the data space for calculating the statistic.
#614 Refactor data classes (fix #563#627#628)
Sherpa's basic data classes have been refactored and cleaned up to help
facilitating fixing bugs and implementing new features. New tests were added to reduce
the chances of introducing regressions in the future.
#612 Fix #609 User Model: unable to change parameter values
A regression introduced in Sherpa 4.10.0 presented users from change
user-model parameter values through direct access. This issue has been
fixed. Several tests were added to reduce the chance of regressions in
the future.
Fix #514 - the command line tool sherpa_smoke in the conda package now correctly honors
command line arguments.

Sherpa 4.11.0 release (rather belated)

Sherpa 4.11.0 has been release for quite some time now, what with a release date of 20 February 2019, and I obviously only just noticed as I came to write the 4.11.1 release blog. Oops. Anyway,on with the show: Sherpa is available via our Conda channel (https://conda.anaconda.org/sherpa/), pip (https://pypi.python.org/pypi/sherpa/4.11.0), Zenodo (https://zenodo.org/record/2573885), and GitHub (https://github.com/sherpa/sherpa/tree/4.11.0).

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, questions, cookies, and any other positive interaction. Please join in the fun on GitHub: https://github.com/sherpa/sherpa
The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.11.0/

OVERVIEW

Optimization routines and statistic methods were made more robust with several bug fixes and improvements.
This version of Sherpa has been tested with Python 2.7, 3.5, 3.6, and 3.7. Support for Python 2.7 is deprecated and will be dropped in a future release.
This release also provides support for XSPEC 12.10.1 (patch 'a' or later) in addition to versions 12.10.0 (included in CIAO 4.11.0) and version 12.9.1.

Details

Documentation and infrastructure fixes are not shown.
#444 Improve caching

When fitting multiple datasets simultaneously Sherpa now tries to cache
model evaluations to improve performance.
#465 Support XSPEC 12.10.0 (fix #436)
XSPEC 12.10.0e is now supported
#508 Ensure fields are initialized before use (pileup code)
Resolved an issue where some fields in the C++ extension could have been
uninitialized in a corner case.
#523 Ensure the ui modules have a unique list of exported symbols (fix #502)
Remove the sherpa.utils.erf symbol from the two ui modules as it is
"over-written" by the ModelWrapper-created symbol when applied to
sherpa.models.basic.Erf.
#524 Be explicit about 'invalid escape characters' in strings
Convert strings that contain "invalid" escape characters, which newer
versions of Python complain about, to raw strings by preceding them with a 'r'
character.
#525 Fixed based line radpro_dm test and changed double --> real (template)
#530 Ensure XSPEC 12.10.0 uses ATOMDB 3.0.9 by default
Ensure that the default AtomDB version is 3.0.9 when using XSPEC 12.10.0.
This is to fix an issue with the models-only XSPEC installation in XSPEC
12.10.0 which uses a default version of 3.0.7 but only provides data files for
3.0.9. Without this change models such as XSapec would return all 0's (unless
the AtomDB version was set manually by the user).
#534 XSPEC 12.10.1 support and mtable fix
Sherpa now supports XSPEC 12.10.1. Several issues were fixed to ensure
compatibility with this release. Note that there are known issues with version
12.10.1 (no patches), so at least v12.10.1a must be used.
#537 / #552 sherpa.rc changes
Sherpa does not read the verbosity.level setting from sherpa.rc anymore but
the option is still in the file for backward compatibility
#539 Fix XSPEC models: mtable table models and ismabs (fix #535#538#540)
Fix to address issues with XSpec- #535 XSPEC multiplicative table models
(mtable models loaded with load_xstable_model); #538 incorrect evaluation of
the ismabs model; #540 the kerrd model evaluated to 0 for XSPEC 12.10.0 and
later.
#546 Small code cleanup in LevMar
small code cleanup in the LevMar class to match the other classes in that
file.
#564 / #567 Harmonise the handling of dof is zero or negative in fit and calc_stat_info (fix #565)
The fit and calc_stat_info methods now use the same code to calculate
the reduced statistic (rstat) and "quality of fit" (qval) values. This
avoids a TypeError when the number of degrees of freedom is negative in
calc_stat_info and ensures that NaN is returned for both values when the
goodness-of-fit calculation has failed (e.g. the reduced statistic is zero or
less or the statistic is negative), whereas in previous versions the qval
value could be None or 1 depending on the code path.
#576 Fix regression with responses get_x method
Version 4.10.1 introduced a regression which made the get_x() method fail
when called on RMF and ARF objects. This has been resolved.
#583 Update pager code (fix #445 #561)
Replace the use of an external pager (such as more or less) with a
Python one. This means that Sherpa no-longer uses the PAGER environment
variable, and that the screen output from the show_* series of commands
should now appear in the correct location when using a Jupyter notebook or the
spyder application.

Friday, December 14, 2018

Sherpa 4.10.2 bugfix release

Sherpa 4.10.2  has now been released and is available via our Conda channel (https://conda.anaconda.org/sherpa), pip (https://pypi.python.org/pypi/sherpa/4.10.2), Zenodo (https://zenodo.org/record/2275738#.XBPp81WYUeM), and GitHub (https://github.com/sherpa/sherpa/tree/4.10.2).

This release supports Python versions 2.7, 3.5, 3.6, and 3.7.

We thank everyone who contributed to this release with feature requests, bug reports, testing, code
contributions, or questions.  Please join in the fun on GitHub: https://github.com/sherpa/sherpa/

The documentation for this release can be found at https://sherpa.readthedocs.io/en/4.10.2/

OVERVIEW

This release fixes a regression introduced in Sherpa 4.10.1 that was noted by the GammaPy team. As part of the long term PSF rebinning improvements, Sherpa 4.10.1 introduced a check to validate that the data pixel size and the PSF pixel size match. If they don't match, then a warning is issued. The change did not account for an edge error case. Sherpa 4.10.2 solves this problem.

Details

#551 Fix gammapy/gammapy#1905, catch errors with psf pixel size.

Sherpa did not properly catch errors when checking the PSF pixel size matches the image pixel size. This has now been fixed.

Caveats

There is a potential issue for macOS conda users with python > 3.6 and matplotlib v3.0.1, when pyqt5 is not installed and selected as a matplotlib backend. With this specific combination of platform and dependencies, matplotlib can fail to work properly (rhe issue can be replicated without importing Sherpa, so this is not a Sherpa bug).

Several workarounds exist, the simplest of which are:

  • install pyqt5 (conda install pyqt`).
  • downgrade matplotlib to v2 or v3.0.0 (conda install matplotlib=3.0.0)


Thursday, October 18, 2018

Sherpa 4.10.1 release

Sherpa 4.10.1  has now been released and is available via our Conda channel (https://conda.anaconda.org/sherpa), pip (https://pypi.python.org/pypi/sherpa/4.10.1), Zenodo (https://zenodo.org/record/1463962#.W8jkmCSYUeM), and GitHub (https://github.com/sherpa/sherpa/tree/4.10.1).

This release supports Python versions 2.7, 3.5, 3.6, and 3.7.

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, or questions.  Please join in the fun on GitHub: https://github.com/sherpa/sherpa/

Overview

This release fixes several bugs and introduces a few new features, notably the ability to evaluate model components on arbitrary grids and generate user-defined ARFs and RMFs. Also, as of this release Sherpa will no longer rely on any Fortran code. See the following section for details.

It is now possible to build the Sherpa documentation using Sphinx. Additionally, the Sphinx documentation is automatically built and hosted on ReadTheDocs. The version for 4.10.1 can be found at https://sherpa.readthedocs.io/en/4.10.1/

Details

#407  Ensure 0-length array is an error in filter_resp (fix #405)

Add an explicit check in the C++ filter_resp code to error out if the noticed channels array is empty.

#422 Improve error message with wrong xspec version

Improve the handling of XSPEC versions mismatch.

#466 Fix bounding box out-of-bounds memory read

Avoid an out-of-bounds memory read when calling pad_bounding_box (when the data is not matching the expected conditions for this call).

#469 Evaluate model on finer grid

Sherpa users can now define arbitrary grids, called evaluation spaces, on which to evaluate individual model components, both in 1D and 2D. This can be useful in a number of cases, for instance when it is desirable to evaluate models on a finer grid than the one defined by the data, or in convolution models where information outside of the data range can be used to reduce boundary effects or to inform the evaluation inside the data space. 

Also, when plotting individual 1D model components (plot_source_component), if a specific evaluation space was attached to the model then the plot will be of the model evaluated over that evaluation space, not the data space. Other plotting commands should be unaffected by this change.

#470 Order for np.ones() should be a one length string not a bool

Fixed a DeprecationWarning in the optimization module.

#471 Using overwrite rather than clobber of astropy.io.fits

The clobber option in astropy has been deprecated for a while now in favour of overwrite and thus issues a DeprecationWarning. Sherpa now uses overwrite instead.

#475 Fix unecessary runtime warning (fix #402)

A condition check in the optimization code was modified so as not to produce a warning when the value is a NaN.

#481 Remove Fortran code

Sherpa does not rely on any Fortran routines anymore, except those compiled in XSPEC. The existing Fortran code, which was used mainly in some of the optimization routines, has  been replaced by equivalent C++ code. This means that gfortran is no longer required for building Sherpa, although a version of libgfortran compatible with the one used to link the XSPEC libraries would still be needed at runtime if the XSPEC extension is built.

#482 ARF/RMF creation functions

The create_rmf and create_arf functions now allow users to easily generate user defined response objects.

#486 ZeroDivisionError in calc_stat_info (fix #476)

Sherpa did not catch divisions by zero in the calc_stat_info command. That has been fixed.

#484 Equivalent Width errors

Several optional arguments (params=None, error=False, otherids=(), niter=1000), were added to eqwidth. If error = True, then get_draws shall be run if the fit stat is one of the following: (Cash,  CStat, WStat) otherwise a multivariate normal distribution shall be run to generate the samples. The optional niter parameter is used to generate the number of samples. The optional parameter otherids is only used if get_draws is used internally when multiple data sets were used to fit. Alternatively, the user can enter the samples via the params option where the samples must be a numpy.ndarray of dimension (num, npar).

#487 PSF bin size warning

Sherpa assumes that the PSF image and the data have the same pixel size. When this is not true Sherpa ignores the difference, which results in a larger PSF being applied. From now on Sherpa will issue a warning when the PSF image pixel size and the data image pixel size are different.

Thursday, May 24, 2018

Reasons to update to Sherpa 4.10.0 (fun with conda)

A quick note about our conda packages for Sherpa

It looks like the conda environment has changed again, which means that the Sherpa 4.9.1 (and earlier) packages we provide for Sherpa may stop working. At least if you are using macOS with Python 3.5 or Python 3.6.

An example of the problem

On macOS (any version), create a new conda environment with sherpa 4.9.1 and the latest version of python:

$ conda create -n sherpa-macos-issue -c sherpa \
          python=3.6.5 sherpa=4.9.1
$ source activate sherpa-macos-issue

At this point, trying to load a Sherpa module will result in the following:

$ python -c "from sherpa.astro.ui import *"
Fatal Python error: PyThreadState_Get: no current thread

Abort trap: 6

This is obviously "not good"™.

This does not appear to be a problem for people using Python 2.7, but we suggest updating to Python 3.5 (or later) as we will be dropping support for Python 2.7 in the not too-distant future.

Solutions

The preferred solution is to update to Sherpa 4.10.0, since they were built with a recent conda installation, and so do not suffer this problem.

$ conda install -c sherpa sherpa=4.10

For users with a source build of Sherpa within a conda environment, remove the original build before re-building, or switch to the binary install above.

If upgrading is not possible, then downgrading the Python version appears to work - that is, use Python version 3.6.1 or 3.5.3.

$ conda install python=3.6.1

$ conda install python=3.5.3

Please let the Sherpa team know, via the Sherpa GitHub page, if there are problems with either of the above approaches.

Happy fitting and modelling!

Tuesday, May 15, 2018

Sherpa 4.10 Release

Sherpa 4.10.0  has now been released and is available via our Conda channel (https://conda.anaconda.org/sherpa), 
pip (https://pypi.python.org/pypi/sherpa/4.10.0), 
Zenodo (https://zenodo.org/record/1245678#.WvsNVlNAnUI), 
and GitHub (https://github.com/sherpa/sherpa/tree/4.10.0). 
This release supports Python 2.7, 3.5, and 3.6.

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, or questions.  Please join in the fun on GitHub: https://github.com/sherpa/sherpa/

OVERVIEW

Sherpa 4.10.0 fixes several bugs related to the support of instrumental responses, including improved support of XMM and Swift responses. Also, this release fixes a significant bug in the support of user statistics, improvements to the Python 3 compatibility, more robust usage of the numpy API, as well as several other minor bug fixes and new tests.
Additionally, this release introduces support for XSPEC 12.9.1n models, as well as the ability to use aliases for parameter names. Some parameter names have been deprecated and may be removed in a future release. We reviewed the parameter limits for many models and updated them to reflect the latest XSPEC specification. Also, multiple versions of XSPEC are now supported, through optional models and version-dependent low-level function calls. This feature is for advanced users building Sherpa from source. Note that Sherpa has been tested against XSPEC 12.9.0i, 12.9.0o, and 12.9.1n. Note that XSPEC is not directly supported by the standalone binary builds, and users are expected to build Sherpa from sources if they want to link it against their version of XSPEC. These changes make it easier for user to link different versions of XSPEC with the same Sherpa code base. Also note, however, that XSPEC 12.10 is not currently supported.
Sherpa now requires pytest-3.3 or later for running the tests.

Details

#451 Release/4.10.0
Testing code now works with pip 10 as well, despite a change in the pip 10 API. Also, the README is now properly rendered as markdown in PyPI.
#438 Change from error to warning for OGIP violations
Given that users can not easily change a response file, and previous versions of Sherpa would allow the responses to be used, this commit changes some of the errors recently introduced (in PR #383) into warnings. The errors for the first bin edge being <= 0 are still left in because users of the sherpa.astro.ui module will find that these files are auto-corrected for them (by PR #383).
#430 Update XSPEC parameters
XSPEC model parameter default values, limits, and properties were reviewed and updated to reflect changes in or mismatches with the model.dat file shipped with XSPEC 12.9.1n.
#428 handle function name changes in XSPEC 12.9.1
Sherpa now supports multiple versions of the XSPEC models from the 12.9.0 and 12.9.1 series. Some models recommend using the C-style interface over the older FORTRAN one, which may also resolve some memory access issues. For CIAO 4.10 users this means the interfaces to XSPEC models have been updated to the 12.9.1n versions. For Standalone Sherpa users this means they can build and run Sherpa against a larger range of XSPEC versions, and Sherpa will pick the XSPEC low level model function accordingly. Note that Sherpa has been tested against XSPEC 12.9.0i, 12.9.0o, and 12.9.1n.
The 14 models that have been changed are: apec, bapec, bvapec, bvvapec, gaussian, lorentz, meka, mekal, raymond, vapec, vmeka, vmekal, vraymond, and vvapec.
#427 Add support for XSPEC models added in 12.9.1 (fix #331)
Add support for models added in XSPEC 12.9.1: bvtapec, bvvtapec, tapec, vtapec, vvtapec, carbatm, hatm, ismabs, slimbh, snapec, TBfeo, TBgas, TBpcf, TBrel, voigt, xscat.
Version 12.9.0 of XSPEC can still be used, in which case the models can be generated - e.g. a user can say xstapec.mdl to create a Python object - but it will error out when evaluated.
The Si and S elemental parameters for the ismabs model have been renamed from the XSPEC versions since they are not unique using the caseinsensitive matching used by Sherpa: so SI, SII, SIII and SiI, SiII, SiIII have been renamed S_I, S_II, S_III and Si_I, Si_II, and Si_III respectively.
Low level support for the following convolution models from XSPEC 12.9.1 has also been added, but there is no Python model support for these: clumin, rfxconv, vashift, vmshift, xilconv.
#412 support ROSAT PSPC (and similar) RMF files with AstroPy
Add explicit tests for reading in, and using, ROSAT PSPC PHA and RMF files. Fix a bug in the AstroPy back-end handling of the ROSAT RMF file.
#409 EmissionGaussian model when skew parameter is not unity (fix #403)
The sherpa.astro.optical.EmissionGaussian code has been fixed for the case when the skew parameter is not unity. The documentation has also been updated.
#399 Python 3 fixes and new tests for pha2 datasets
Fixed several problems when using Sherpa with Python 3: failures when calling list_bkg_ids and list_response_ids and the parameter querying after paramprompt(True) has been called.
There is also a change to avoid a FutureWarning being raised in the astropy backend when reading in PHA2 data files.
Tests have been added to test the reading of a PHA2 format dataset, and new files added to the sherpa-test-data repository for this purpose.
#398 Check that list_samplers returns a list (fix #397)
The list_samplers function now returns a list in Python 3 as well.
#396 Fix set_xlog and related commands using Python 3 (fix #393)
Fix use of commands that set the plot state using the sherpa.astro.ui module, such as set_xlogand set_xlinear, when using Python 3.
#395 Add xspec optional models
Sherpa has new infrastructure for supporting multiple versions of XSPEC at the same time, with models that are built and enabled conditionally depending on the version of XSPEC being linked. Models are assumed to have the same parameters and low level functions across supported versions.
#394 XSPEC build and functionality improvements
Add support for reading and changing the path to the XSPEC "manager" directory, and reading the current XSPEC "model" directory (get_xspath_managerset_xspath_manager, and get_xspath_model). These are intended for the power user and so require an explicit import from sherpa.astro.xspec.
There are several improvements to the build and interface to the XSPEC model library: these have no user-visible changes.
#390 Add nlapec model
The nlapec XSpec model has been added. Note that XSPEC 12.9.0 is now required.
#385 DS9 calls return string in Python 2 and 3 (fix #319)
Some low lever xpa calls were returning byte strings rather than strings in Python 3. This had particular impact on the image_getregion function. This has now been fixed.
#383 Replace 0-energy bins in ARF and RMFs (fix #332)
Sherpa now performs some validation of the energy ranges of ARF and RMF files against the OGIP standard. If the ranges are inconsistent then the code will return with an error. If one energy bound is 0 the bound is replaced with a small number to avoid numerical issues like division by zero. A new configuration option minimum_energy, if present, allows users to override Sherpa's default behavior.
#379 Use a line not span to draw horizontal line (fix #378)
Ensure that the line at y=0 (residual) or y=1 (ratio) is drawn for residual or ratio plots when using matplotlib 2.0.
#373 Make sure ds9 properly works with Python 2.7 (fix #368)
For Python versions prior to 3.2 add in explicit code to make _Popen work as a context manager.
#352 Add low-level support for the XSPEC rgsxsrc convolution model
Add low-level support for the rgsxsrc convolution model in XSPEC. This is intended for R&D into fully supporting XSPEC convolution models in Sherpa.
#255 Add aliases for parameter names so to support new xspec names (fix #74)
The XSPEC models have been updated to use parameter names that match the new XSPEC naming scheme introduced in XSPEC 12.9.0. The old parameter names can still be used as aliases for the new name, but are deprecated and may be removed in a next major release. Note this allows any models to define aliases for their parameters.

Friday, August 4, 2017

Sherpa 4.9.1 Release

Sherpa 4.9.1 - an incremental improvement over the Sherpa 4.9 release - has now been released and is available via our Conda channel (https://conda.anaconda.org/sherpa), pip (https://pypi.python.org/pypi/sherpa/4.9.1), Zenodo (http://doi.org/10.5281/zenodo.838686), and GitHub (https://github.com/sherpa/sherpa/tree/4.9.1). This release supports Python 2.7, 3.5, and 3.6.

We thank everyone who contributed to this release with feature requests, bug reports, testing, code contributions, or questions.  Please join in the fun on GitHub: https://github.com/sherpa/sherpa/

Overview

This version introduces full support for Python 3.6. It also fixes issues with non-Chandra response files, correctly handles the AREASCAL column in PHA files, and fixes a significant regression that was preventing user statistics from working in v4.8.2. It also introduces a number of smaller improvements and fixes. In particular, quite a few improvements have been made to the documentation and to the testing framework, including several new tests to improve stability.

Details

Infrastructure and minor non-functional changes have been omitted.

#335 Fix `setup.py` install command

The `setup.py install` command was not enforcing the installation of the dependencies listed in `setup.py`. This has been fixed.

#368 Remove ds9 warnings when run under Python 3.6

Update the DS9 code so that external processes are cleaned up properly, so to remove the potential ResourceWarning warnings when running DS9 on Python 3.6.

#351 fix handling of AREASCAL column in PHA files (fix #350)

Add support for handling the AREASCAL value (either scalar of vector) for PHA data sets. This array is used in XMM RGS data to handle missing chips.

#358 Properly handle Swift RMF when using Astropy backend (fix #357)

A Swift RMF could not be read in when the AstroPy back end was in use. The problem was that the code did not support RMF matrices that were not stored as variable-length arrays. This has now been fixed, and new tests have been added for this kind of file.

#343 Fix user statistics regression (fix #341)

A number of regressions were introduced in version 4.8.1 up to version 4.9.0, so user statistics that were properly working in version 4.7 have not been working any more. This has been fixed, and a number of regression tests have been added.

#381 Fix `bdist_wheel` command

We reviewed our customization of the setuptools commands as we realized that a recent bug fix (#335) introduced a regression in the bdist_wheel command used by pip to build Sherpa from source distributions. Both commands should now properly work.