Source of the Concrete Data

[SmokinCaterpillar]

Hey,

super interesting project. I am just wondering what is the source of the concrete data? Could it be that the dataset was available to chatGPT prior to this study? Could there be some training data leakage?

Thanks a lot and all the best,
Robert

[iterateccvoelker]

Hi Robert,

Thank you for your interest and for raising such an insightful concern! We'll certainly look into it thoroughly. As of now, our concrete data source is: https://doi.org/10.1080/14488353.2018.1450716, which has also been used by other authors: https://doi.org/10.1016/j.conbuildmat.2020.119380. Although it's not impossible that this data was part of GPT's training set, it's worth noting that there are numerous published formulations, making this one relatively insignificant. The improvement in prediction performance due to in-context learning (ICL) suggests that context plays a crucial role.

To initially address your concern, we could consider adding an arbitrary offset (e.g., 453.54564) to the label and re-conduct the benchmarking. Would you find this approach sufficient in masking the data?

Warm regards,

Christoph

[iterateccvoelker]

BTW: training and test sets have been saved for all models (https://github.com/ghezalahmad/LLMs-for-the-Design-of-Sustainable-Concretes/tree/main/results). Each model has been trained and tested on the same formulations, albeit ICL has reformulated the compositions as text. Please note that this work addresses small data problems (10 training data points). If more data is available I guess that RF outperforms.

[SmokinCaterpillar]

Hi Robert,

Thank you for your interest and for raising such an insightful concern! We'll certainly look into it thoroughly. As of now, our concrete data source is: https://doi.org/10.1080/14488353.2018.1450716, which has also been used by other authors: https://doi.org/10.1016/j.conbuildmat.2020.119380. Although it's not impossible that this data was part of GPT's training set, it's worth noting that there are numerous published formulations, making this one relatively insignificant. The improvement in prediction performance due to in-context learning (ICL) suggests that context plays a crucial role.

To initially address your concern, we could consider adding an arbitrary offset (e.g., 453.54564) to the label and re-conduct the benchmarking. Would you find this approach sufficient in masking the data?

Warm regards,

Christoph

Hi Christoph,

thanks for the quick answer.

An arbitrary offset seems plausible (or maybe just an arbitrary linear transformation so even rescaling it?). I am especially curious if the size of the offset or magnitude of the scaling factor could degrade the performance, as chatGPT has challenges with large numbers. And then also what happens in case of nonlinear transformations? I'm really curious how this effects chatGPT's capabilities.

[iterateccvoelker]

Thanks for the comment, Robert! I am very excited, too. BTW: if you want to fast-lane some results: it is possible to copy-paste the prompts into ChatGPT. It is not the identical model and settings so the answer might vary but it should be somewhat similar.

You can find the finetuned ICL prompts here: https://github.com/ghezalahmad/LLMs-for-the-Design-of-Sustainable-Concretes/blob/main/results/ICL_finetuned/train.txt

And the vanilla ICL prompts here: https://github.com/ghezalahmad/LLMs-for-the-Design-of-Sustainable-Concretes/blob/main/results/ICL/train.txt

an example could look like this:

We will do an exercise where I will provide you with concrete formulations as prompts and their respective respective compressive strength as completions for you to learn from. Then you will only receive prompts and need to complete it yourself. Add the respective Idx to each answer. Lets go:

prompt: "Idx 65, The concrete consists of Powder weight: 420, Liquid Weight: 210.0 with a Liquid/Powder-Rate of 0.5, 16.8 kg Plasticiser and 1754.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.5/0.5, Heat curing (Rao et al.)" ; completion: 64.28
prompt: "Idx 88, The concrete consists of Powder weight: 380, Liquid Weight: 171.0 with a Liquid/Powder-Rate of 0.45, 15.2 kg Plasticiser and 1833.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.6/0.4, Ambient curing (Rao et al. 2018)" ; completion: 43.58
prompt: "Idx 128, The concrete consists of Powder weight: 380, Liquid Weight: 171.0 with a Liquid/Powder-Rate of 0.45, 15.2 kg Plasticiser and 1833.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.6/0.4, Heat curing (Rao et al.)" ; completion: 56.66
prompt: "Idx 43, The concrete consists of Powder weight: 360, Liquid Weight: 216.0 with a Liquid/Powder-Rate of 0.6, 14.4 kg Plasticiser and 1808.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.5/0.5, Heat curing (Rao et al.)" ; completion: 52.4
prompt: "Idx 213, The concrete consists of Powder weight: 390, Liquid Weight: 195.0 with a Liquid/Powder-Rate of 0.5, 15.6 kg Plasticiser and 1799.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.7/0.3, Heat curing (Rao et al.)" ; completion: 43.03
prompt: "Idx 69, The concrete consists of Powder weight: 430, Liquid Weight: 215.0 with a Liquid/Powder-Rate of 0.5, 17.2 kg Plasticiser and 1739.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.5/0.5, Heat curing (Rao et al.)" ; completion: 63.47
prompt: "Idx 106, The concrete consists of Powder weight: 420, Liquid Weight: 231.0 with a Liquid/Powder-Rate of 0.55, 16.8 kg Plasticiser and 1733.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.6/0.4, Ambient curing (Rao et al. 2018)" ; completion: 44.67
prompt: "Idx 1, The concrete consists of Powder weight: 360, Liquid Weight: 180.0 with a Liquid/Powder-Rate of 0.5, 14.4 kg Plasticiser and 1844.0 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.5/0.5, Ambient curing (Rao et al. 2018)" ; completion: 59.79
prompt: "Idx 214, The concrete consists of Powder weight: 390, Liquid Weight: 214.5 with a Liquid/Powder-Rate of 0.55, 15.6 kg Plasticiser and 1779.5 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.7/0.3, Heat curing (Rao et al.)" ; completion: 32.58
prompt: "Idx 116, The concrete consists of Powder weight: 450, Liquid Weight: 202.5 with a Liquid/Powder-Rate of 0.45, 18.0 kg Plasticiser and 1731.5 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.6/0.4, Ambient curing (Rao et al. 2018)" ; completion: 45.88

What is the completion of: prompt: "Idx 46, The concrete consists of Powder weight: 370, Liquid Weight: 203.5 with a Liquid/Powder-Rate of 0.55, 14.8 kg Plasticiser and 1810.5 Aggregates. The powder constituents are Fly Ash/GGBFS-Ratio: 0.5/0.5, Heat curing (Rao et al.)"

You can even invert the problem and say: give me a prompt for completion: 45.00

It is possible to define the boundaries of the solution, too. For example, the minimum Liquid/Powder-Ratio should be 0.4. Or you can be more abstract and say give me a prompt (mixture) that meets DIN-EN XXXX.