Timeseries object for Astropy (APE9)

[Cadair]

This is still in very early draft.

[abigailStev]

Pinging myself so I can follow development and discussion.

[ehsteve]

I am very interested in this. Thanks for putting this together @Cadair.

[ehsteve]

and thanks to everyone that participate in the discussion at PyAstro16 :P

[ehsteve]

remove the goal of sharing. that's a general astropy goal.

[ehsteve]

I think it would be helpful to list a number of examples of analyses that we would want to perform on these.

[duncanmmacleod]

I am interested in this feature, and think it would be a great addition. GWpy would likely move to using/inheriting this object, rather than the current gwpy.timeseries.TimeSeries (which inherits from astropy.units.Quantity in the first place).
I'm happy to help with development and testing as desired.

[abigailStev]

I am in support of this APE! And I'm available to help with testing.

[eteq]

I like it and would use it.

[eaydin]

I love it!

[abigailStev]

I would like for @StingraySoftware to move towards using a TimeSeries object (or at least basing our Lightcurve class on it) for lightcurves.

[SolarDrew]

I think we decided this was not going to be necessary.

[SolarDrew]

How many of these features already exist in QTable?

[Cadair]

Some, but all would need to be customised for timeseries data (i.e. to preserve the index correctly.)

[Cadair]

Re-word to clarify that concatenation is extra columns.

[Cadair]

Should we be able to construct from a start time and a list of "deltas"?

[Cadair]

This section should also demonstrate constructing these objects with Time and TimeDelta so that it's clear that's a thing.

[Cadair]

and missing value (NaN)

[mwcraig]

I'm late to this, but have a couple things to add:

• I like the TimeSeries/BinnedTimeSeries option more than the original Sampled... and Binned...
• I'd propose (this may be a little extreme) that in the BinnedTimeSeries there not be a time property at all. Instead, I'd go with the more explicit start_time, end_time and mid_time (or something along those lines). It is a little more verbose but has the virtue of being very clear what time you were using without needing to track down the documentation.

[gbelanger]

Good morning,

I just read through this thread and find the discussion very useful and important. I have implemented, over the past 12 years, a bunch of packages for time domain analysis in Java, and have had to make all the choices that are being discussed above in terms of how to represent time-related data.

Because my background is high energy astrophysics, it was clear from the very start that I needed to have EventList and TimeSeries as different objects. As it was mentioned above, EvenList can have other characteristics in addition to arrivalTime such as x, y, energy, and a quality flag, for example. In the TimeSeries this is not the case. But there can be fundamentally different kinds of time series, like what I've called CodedMaskTimeSeries that carries a lot more information than just intensity as a function of time, because the time series is actually reconstructed from images, and the errors depend on the angle at which the source was in the field of view, and on and on.

So, I have a package for event lists and a package of time series, and in the time series package, I have an interface ITimeSeries, and abstract class AbstractTimeSeries, and two concrete classes TimeSeries, and CodedMaskTimeSeries going from the general to more specific, each class inheriting from the previous and each implementing the interface. I also have an interface ITimeSeriesFileReader with specific file readers that implement it for different kinds of input files (fits, ascii, qdp), such that the TimeSeriesReader just loops through to try each implemented type when given an input file. Similarly, there are several TimeSeriesFileWriters.

The similarity between an EventList and TimeSeries in terms of its temporal information carrying capacity is used in the factory classes that can handle time data: the TimeSeriesMaker and PeriodogramMaker. These have methods to deal with event lists and with time series as inputs. For the periodograms, it is again with an AbstractPeriodogram that is used as the base, and then different types that extend it (FFTPeriodogram, RayleighPeriodogram, LombScarglePeriodogram, ZPeriodogram, ModifiedRayleighPeriodgram, LikelihoodPeriodogram, as well as AggregatePeriodogram to combine periodograms and AveragePeriodogram to express the result, which is the only periodogram that has uncertainties on the powers, because they are not used in any other one since the power a each frequency in a periodogram is a point estimate.

Lastly, I have been recently re-writing the resampling of time series in order to make it more manageable, because dealing with resampling of a binned time series in a general way is very complicated. My time series are allowed to have both irregular sized bins and irregular spaces in between them, and everything is written to handle that. So, I have been working on decomposing the time series into a collection of intensity bins made up of (composition) an Intensity and a Bin, each of which carries specific information that defines everything we need to know. In this way, we can handle each IntensityBin one at a time. This is ongoing and I'm still sorting out difficulties like how to deal with different kinds of intensities (absolute quantity or density, for example) in a general manner.

But in conclusion, I think that firstly, having EventList and TimeSeries objects makes sense, because an EventList is not really a kind of TimeSeries even though its arrival times can be used to make one. And secondly, it is very important to look down the line and consider all the kinds of things that will be done with the EventList and TimeSeries (e.g. periodograms or resampling) so that it doesn't become too difficult to do them when we get there. Please let me know if you want more details about anything addressed in this comment. I'd be happy to help further. Unfortunately, I haven't yet started coding in python. Maybe it's the time to start ;) I also need to update my repos on GitHub which I promise to do soon.

[astrofrog]

@gbelanger - thank you for your insight - it's always really helpful to know that people have gone through similar design questions in the past and see how they have solved them. I think it would be really helpful if you take a look at what we have so far (once ready to try out) and see if you have any suggestions (and it would be great if you were interested in contributing!)

[astrofrog]

I've now had a chance to digest all the suggestions above and have now updated the prototype implementation. I've also now put the implementation in a standalone package to make it easy to try out. The repository and docs are here:

https://github.com/aperiosoftware/astropy-timeseries
📖 https://astropy-timeseries.readthedocs.io/en/latest/

It would be really helpful if people here can take it for a spin to see if you like the current API. The way I've set up this package is such that once we open a PR to astropy core, all commits will be preserved. Therefore, feel free to open pull requests, or issues! If you'd like to work on some of the open issues, just leave a comment!

You can easily install the package with:

pip install git+https://github.com/aperiosoftware/astropy-timeseries.git

and note that things will work better if you have the latest developer version of Astropy (or the 3.1rc1).

A few updates based on the above discussion:

• I've decided to stick with separate classes for the binned and sampled time series for now, but calling the sampled one TimeSeries rather than SampledTimeSeries

• TimeSeries has a time attribute, while BinnedTimeSeries instead has time_bin_start, time_bin_center, time_bin_end, time_bin_size attributes to make sure everything is explicit.

• I've moved some of the functionality (e.g. the downsampling) out of the base class since the implementation that's there may not be appropriate for all cases. In general I think the base time series classes should be as generic as possible, and we or affiliated packages can then implement sub-classes if needed.

• I don't think we need to take a stance on event lists - if people want to use TimeSeries for event lists, they are free to do so, but some packages may want to implement their own specialized event list class.

Let me know if you have any feedback!

[StuartLittlefair]

I'm a little late to this party, but having looked at and tried the prototype I have a suggestion which might make the TimeSeries class more powerful. Unfortunately I think it would mean a quite different implementation.

If I imagine my use cases for a class, it is to handle collections of data, which have a matching time attribute. Typically, data has values, uncertainty, and perhaps a mask indicating data quality - just like the NDData class.

It would also be incredibly powerful if arithmetic on TimeSeries objects performed arithmetic on matching "columns", whilst handling the uncertainties and masks automatically. Showing, an example, wouldn't it be great if something like this worked...

flux_1 = NDDataArray(data=y_1, uncertainty=StdDevUncertainty(yerr_1), flags=flags_1) 
flux_2 = NDDataArray(data=y_2, uncertainty=StdDevUncertainty(yerr_2), flags=flags_2) 
ts_1 = TimeSeries(time=t, data={'flux': flux_1})
ts_2 = TimeSeries(time=t, data={'flux': flux_2})
rel_ts = ts_1/ts_2

...and the rel_ts object also had a flux entry which had the correct values, uncertainty and mask? Users could subclass NDDataRef to get have the flags combine anyway they'd like.

Such behaviour is compatible with the APE as it stands, but can't be achieved if a TimeSeries is a subclass of Table, since they don't support having NDDataArray as a column, as far as I'm aware. One would lose the ability to join, stack etc - but I'd argue that the functionality above is more powerful for many users. I've posted a highly incomplete prototype to illustrate how it might be implemented here.

Sorry if this suggestion comes after many people have done a lot of hard work; I hadn't realised this issue until I started to play with the implementation. Feel free to ignore this, or point out glaring errors

[astrofrog]

@StuartLittlefair - thanks for the suggestion! I agree that using NDData might work well for some use cases. I'd like to make two suggestions:

• In specutils, we made a spectral mix-in class that can be used to make one of the NDData dimensions be a spectral axis. We could do the same for time, i.e. make a mix-in class that provides an easy way to set up one of the axes of an NDData cube as a time axis. Note that this would then also allow one to set up cubes with two spatial, one spectral, and one time axis. To me, this is complementary to the Table approach - in some cases people want to think of time series as being simple tables, and in others people might want to think of it as an n-dimensional cube (where n>=1) where one axis is time. (Implement time series mix-in class for NDData aperiosoftware/astropy-timeseries#16)

• There's no reason we can't make NDDataArray be used as a mix-in column in tables - in fact I'll open an issue about that now! (Make it possible to use NDDataArray as a mix-in column in tables astropy#8159)

I think both of these can be done of course, we don't have to choose.

I'll also mention that for NDData, we changed the API a few times because we ended up making the base class try and be too smart, and some functionality is best left to sub-classes. I think we will need to do the same here - that is, functionality like rel_ts = ts_1/ts_2 makes too many assumptions to belong in a base time series class. However, we could always develop a sub-class that behaves in the way you indicate, as a convenience layer.

I'll try and work on an NDData mix-in class implementation to see how much work it might be and whether it would suit your use case. I'll also see if I can re-use some of your prototype code.

[gully]

Should we merge this APE since astropy.timeseries is merged?

[bsipocz]

@gully - I would rather wait until 4.0, and ask everyone (especially folks with non single band optical photometry use cases) here to give the current version a good try and report back issues or missing functionalities.

[gully]

Thank you @bsipocz , indeed learning from the existing timeseries implementation should feed back to the long term vision.

[bsipocz]

(well, it's not my call to anyway to merge this, but the feedback I heard so far is that the current api is very much kepler type light curve centric. but it's very very difficult to anything about it without actual issues being opened...)

[gully]

Interesting, I should try out the timeseries class more to see what feedback I can give. I am curious about multichannel and sparse applications too, despite being Kepler-centric much of the time. I'm especially interested in use cases that combine sparse ground-based observations with high cadence space based observations. By "combine", I don't necessarily mean in the same "flat" timeseries object, though that's conceivable. I just mean making sparse and high cadence interoperate more easily would unlock some good science. The different bandpasses comes up a lot too.

I re-read the APE governance protocol on the README and appreciate that a lot goes into formalizing the APE, so my use of "merge" wasn't the right choice. I had thought an APE was a pre-requisite to making new functionality, but I see now that making an experimental implementation further informs use cases and guides the functionality in the longer term.

[bsipocz]

@gully - make sure you're playing with the dev version as there are some substantial improvements in the underlying classes that were necessary to be able to extend the functionalities (though none of those are reflected in the narrative docs yet)

[Jerry-Ma]

Hi,

I am working on the TolTEC project and I am wring python code for handling the time stream data collected from an array of bolometers that scans in time.

As already pointed out in the previous comments, since each time sample contains tightly packed data (in our case, a vector of size ~4000, matching the total number of detectors in the array), it would make most sense to have an NDData as the storage and have timeaxis mixin attached to one of the axis.

I saw that #16 is still open and I was wondering what is the status of that?

[pllim]

So... astropy/astropy#8540 was merged and released in v3.2. Where are we going with this in 2022?

[pllim]

Pinging again in 2023. Are we just going to wrap this up since timeseries is already in core and all that?

[astrofrog]

We should have a "Superseded by events" status for APEs 😅 (jk!)

[gully]

So is this APE complete now that timeseries exists? I guess a repeat of the question above. Would folks say that the aspirations of this APE followed through in the implementation?

[pllim]

Would have to ping @astrofrog and @Cadair again

[hamogu]

@eteq Please follow through with the Zenodo steps when they are online again (they had trouble minting DOIs the last few days).