Google Summer of Code 2025: Quartz Solar - New data source in ML model Discussion Thread

[AUdaltsova]

This space is for you to ask any questions you have about this project. We're here to provide clarifications and help you understand the project's goals, scope, and requirements. Feel free to ask about anything that interests you!

Please note that this discussion is for questions and clarifications, not for formal applications.

Project Description

Adding new data sources usually gives a boost to the predictive power of our models, and finding innovative ways of extracting information from them can be even more beneficial. We would like to explore ways to improve our solar energy forecast with an ablation study of how much data on features like dust or neighbouring sites can contribute to the precision of the model. The project can be scaled depending on time constraints.

Expected Outcome

A comparative analysis of the effects of auxiliary data sources on the forecast quality.

Other Key Information

• Expected Size: 175hrs
• Skills: Familiarity with training ML models, preferably with pytorch. Data analysis skills would be highly beneficial
• Difficulty level: Medium-Hard
• Related Reading: https://github.com/openclimatefix/open-data-pvnet, https://github.com/openclimatefix/PVNet
• Potential mentors: @AUdaltsova, @Sukh-P

[celestinelindarto]

Hi @AUdaltsova @Sukh-P,

Thank you for organising this year’s event and putting this document together for the GSOC program. I am interested in your project and would like to know further details.

• Are the additional features such as dust and neighbouring sites supplied from Open Climate Fix or will it be sourced from an open-source database?
• What metric is used to measure the performance of a model?
• What model(s) is/are primarily needed for this project?
• I noticed that the difficulty level for this project is medium-hard and the project timeline can be scaled accordingly as per the project description above. Does it mean that the project may be upgraded into a large-sized project (according to Google Summer of Code’s classification)? Is it possible to discuss the length of time required to complete the project?
• Is there any ongoing issues with this particular project that I can potentially evaluate and contribute to?

I look forward to hearing from you.

Thank you for your time.

[AUdaltsova]

Hi @celestinelindarto, thanks for your interest!

• I expect all new data will need to be sourced from open-source providers. We might have some related download scripts if we've worked with that provider before
• We usually use MAE for performance assessment
• Ideally, you would use PVNet (v4, so with ocf-data-sampler as the batch-creating pipeline), so the results are immediately comparable to our previous work. There might be a couple of pain points with this, as PVNet's quite large and requires a lot of data to train, so it might not be feasible to run locally - we could provide access to our servers instead to make that work. Alternatively, this would also work as a proof-of-concept project on a relatively small model you put together yourself instead, if we decide it's easier than trying to use PVNet
• In terms of difficulty and intended time frame, this was meant more as "we can be flexible with what to explore and how many options to try". I expect, lets say, 'minimum scope' version of this (testing one hypothesis - one new source of data or one new way of processing existing data) to fit well into a medium-sized project; depending on how fast you work, this time frame can fit more experimentation, or we can agree to extend to a 'large' project. If you have concerns on fitting all the work into a medium project, we can also increase the timeline.
• I don't think we have any ongoing issues related to this unfortunately, but I think open-data-pvnet could be a decent starting point, as a lot of the work done there on plugging in new sources into PVNet would be relevant here

Hope this is helpful! Let me know if you have other questions

[yuvraajnarula]

Hi @AUdaltsova and @Sukh-P,

Thank you for sharing this fascinating project on improving solar energy forecasting models. Building on the discussion so far, I'd like to offer some additional thoughts and questions:

Regarding auxiliary data sources:

• Have you considered satellite-derived aerosol optical depth measurements as a proxy for dust/particulate matter? This might provide more consistent global coverage than ground stations.
  What spatial resolution are you targeting for neighboring sites? I've worked with models that benefit from incorporating data at multiple scales (1km, 5km, 10km radii).

On the technical approach:

• Is there interest in exploring transformer-based architectures that can effectively capture the temporal dependencies in solar irradiance patterns while incorporating spatial correlations?
• Would you prefer traditional ablation studies or more interpretable methods like SHAP values or integrated gradients for feature importance analysis?

I have experience implementing similar forecasting models with PyTorch, particularly focusing on:

• Multivariate time series forecasting with environmental variables
• Transfer learning techniques to adapt models across different geographical regions
• Uncertainty quantification in forecasts, which could be valuable for energy grid planning

I'd be happy to elaborate on any of these approaches or discuss other aspects of the project that might be helpful. The scaling aspect is particularly interesting - determining how generalizable these models can be across different climate zones.

Looking forward to learning more about this project!

[AUdaltsova]

Hi @yuvraajnarula, thanks for your interest! Some great questions here :)

Regarding auxiliary data sources:

• Aerosol optical depth is definitely on the table; we've done some experimentation on it before (see here) - it was looking promising at first, but we had trouble extracting performance in training. It could definitely be revisited! We were also looking at combining satellite channels in the data we already use to extract dust proxy, like EUMETSAT does here. I'd say we would generally aim for accessible data with consistent coverage, so definitely satellite observations over ground stations.
• I don't think we've had a specific spatial resolution in mind for the neighbors, so can be something to explore. Do keep in mind that these were just a couple of examples though - while we can definitely explore those, other ideas are also welcome and you don't have to stick to dust and neighbors!
  .

On the technical approach:

• If you're more interested in experimenting with architectures, unfortunately this might not be the project for it - I'd refer you to this one, which is the closest we have in proposals I think (though if I recall correctly it doesn't explores transformers). Here, I think it's best to stick with PVNet and dedicate most of the attention to data sources
• I don't think we have a strong preference on ablation methodology. More interpretability is always great, but I think as long as we can clearly compare performance with and without modifications and, ideally, have an assessment of which variables from the new source contribute to the performance (important as our models and archives tend to be bulky - we'd always prefer to not download extra variables if it won't make much difference!), you're free to use whatever makes most sense.

Your past projects sound super interesting! Re: scaling across different climate zones, that could definitely be an option here as well, as we work in very different parts of the world, so would be great to see how improvements translate between them.

Hope this answers your questions! Let me know if you want to discuss more

[yuvraajnarula]

Hi @AUdaltsova,

Your project's focus on solar forecasting triggered an unconventional thought. Instead of traditional neighboring site analysis, what if we explored solar variability through atmospheric "information propagation" networks?

Imagine mapping solar irradiance dynamics using concepts from network science and information theory - treating meteorological conditions like a complex adaptive system where energy variations "flow" across regions.

Key innovative angles:

• Modeling solar variability as an information transmission problem
• Using entropy-based metrics to quantify predictive interdependencies
• Treating geographical sites as nodes in a dynamical network where energy signals "cascade"

This could offer insights beyond simple spatial correlation by:

• Revealing non-linear relationships between sites
• Capturing how solar variability "ripples" across different climate zones
• Providing a more nuanced understanding of regional interconnectedness

The analysis would be lightweight, focusing purely on extracting meaningful insights for PVNet without architectural complexity.

Would you be interested in exploring how information theory might reframe our understanding of solar forecast data propagation? I'm particularly excited about bringing an interdisciplinary lens to your existing framework.

[AUdaltsova]

Hi again @yuvraajnarula! Sorry for taking a bit to respond. That sounds super interesting and exciting, I'd love to see this come to life. One concern I have though is this fitting into the scope of this project, as it's pretty focused on finding ways for us to improve on our current set-up without conducting a major re-haul, but sounds like you've thought of that as well!

Can please you elaborate a bit on how you imagine this work informing PVNet's forecast so I have a better idea? I have trouble envisioning the set-up.

Also, just for context, off the top of my head, I'd expect we'll be constrained to the UK by the pv data we have access to, but I think there's potential in treating it as one "system" and the data might lend itself quite well to your idea!

[ajitashwathr10]

Hi @AUdaltsova

I am keen on working on this project and thought that the idea was really fascinating. I do have some questions, before proceeding:

How would we structure an ablation study to measure the independent contribution of dust levels and nearby site information to solar energy forecasting performance?

What data preprocessing methods would be suitable for dealing with noisy or missing dust level data before it is used in the model?

How do we adapt the current PyTorch-based PVNet model to include more features without substantially raising computational overhead?

And, what extent of prior experience in data analytics is necessary for effectively contributing to this project?

[AUdaltsova]

Hi @ajitashwathr10! It's great to hear you are interested!

• I'd expect you'll need to train the model without any adjustments, and then a version for each new source and possibly one with both of them combined. This way, we can directly compare performance and clearly see how sources contribute to it. Additionally, you can perform some more complicated assessments, such as feature importance etc
• PVNet is set up to pick up only the samples where the data from all requested data sources is available, so I would expect no interpolation to be involved, and if there are portions of the data that are very noisy they can be dropped as well, but if you have other ideas we can always do something different!
• PVNet currently uses separate encoders for each data source which are relatively easy to add. You're right that it would increase compute, and if a new source supplies relatively little data compared to what usually goes into an encoder, that can cause overfitting, so depending on the data there will be room to experiment a bit and maybe chuck out parts to decrease the number of parameters
• You will be expected to asses any new data that you put into the model so some basic data skills would be really good to have (just on the level of being able to compile data into one dataset, checking value ranges, looking for noise/outliers). It's always better if you have manipulated and cleaned data before and can come up with how to best approach the dataset you end up with on your own, but we will be able to help to some extent. No overly complicated data processing techniques are expected, but general understanding of the features the data has and how that can translate into what comes out of the model would be really good to have (e g, we normalise all data to have 0 mean, 1 std distribution before it goes into the model, so all values end up more or less in the same range; what could happen if the original variable has a really long tail on one side?)

Hope this is helpful! Ping me if you have any more questions

[leoheim]

Hi @AUdaltsova and @Sukh-P,

Thank you for sharing this exciting project! I’ve been going through the details, and I have a few questions to better understand the challenges and potential directions:

Since PVNet relies on a substantial amount of data for training, have you considered any techniques to address potential data limitations, such as data augmentation or transfer learning? Would these approaches be relevant in this context?

Given the goal of evaluating the impact of new data sources, what criteria are used to determine whether a new feature is valuable enough to justify its inclusion in the model? Is there a structured process for feature selection, or is it more experimental and iterative?

When evaluating the impact of additional data, do you typically analyze its effects across different time scales (e.g., short-term vs. long-term forecasting), or is the focus primarily on immediate predictive accuracy?

Given that the project involves assessing new data sources, how do you typically balance adding complexity to the model versus maintaining interpretability? Are there any techniques you prioritize to ensure that new inputs provide meaningful contributions without overfitting?

Have you considered domain adaptation techniques to improve model transferability across different geographic regions or weather conditions? Given that solar energy forecasting can be highly location-dependent, what approaches have been most effective in ensuring model generalization?

I’m really interested in this project and would love to contribute meaningfully. Looking forward to your insights!

[AUdaltsova]

Hi @leoheim, thanks for your interest!

• There can be room for this, but I think the core part of the project could get quite bulky already, so I think we would prioritize sources that already provide enough data coverage rather than trying to incorporate something that has additional challenges. We can try if you have an idea for a really good source that would be worth it, but generally I would say this is outside of the scope
• Main assessment method would be comparing MAE on different forecast horizons. We don't have a rigid selection procedure, it's more iterative. I'm very familiar with the urge to conduct exhaustive experimentation but we might be limited by the project's time frame here :)
• Again, I wouldn't say we have a specific procedure that tells us a particular source is worth it, mostly we look at performance improvements, some example plots, and quantile fitting (PVNet is based on quantile loss and produces several quantiles, typically seven, and we use the median as the point prediction. If the fit of quantiles is off on validation that can be a sign of overfitting/botched normalization/other problems with the model)
• We currently do not transfer the model across different locations and prefer to retrain with specific regional data. I think we will try to generalize at some point and personally, I find it a very promising avenue for research, but it would probably fall outside this project's scope as well

Hope that's helpful! Let me know if you want to chat some more.

[leoheim]

Hi @AUdaltsova ,

Thank you so much for the thorough and insightful response! I really appreciate you taking the time to explain the project’s approach and considerations in such detail. It’s great to gain a deeper understanding of the thought process behind data selection and model evaluation.

If there’s anything I can do to assist or contribute, I’d be more than happy to help. Just let me know!