Update tutorials for 1.0 and automate testing where possible

[sffc]

The tutorials in files such as "intro.md" are woefully out of date. We should make sure they are updated for 1.0, and also automate testing as much as possible, e.g. by moving code samples into a module such as icu::tutorial and using docs tests.

[dsipasseuth]

Do you have a preferred location/structure for the module ?

For now, I'll just put it under component/tutorial on my PR, but happy to move it anywhere it makes sense.

[sffc]

components/icu/src/tutorial.rs would be a good spot. You can make the module #[cfg(doc)] and then it shows up in docs but not elsewhere.

[dsipasseuth]

Do you want to merge all 4 md files from the tutorial folder into one giant tutorial.rs file ?

[sffc]

You could make a directory components/icu/src/tutorials and then different .rs files inside

[dsipasseuth]

There's actually a way to make .md files contain rust code that will be tested just by adding a few directive.

#[doc = include_str!("../../../docs/tutorials/intro.md")]
#[doc = include_str!("../../../docs/tutorials/data_provider.md")]
#[doc = include_str!("../../../docs/tutorials/writing_a_new_data_struct.md")]
#[cfg(doctest)]
pub struct Docs;

Also, I need some help with the "writing a new data struct". I could not manage to get a "simple version" that just compiles and run.
I think it needs a more in depth rewrite. (May be use hello_world.rs as a baseline for a "step by step" and explaining each concept as it move further in the implementation ?)

For now, I've remove the examples code from writing_a_new_data_struct.md and replace them with Github code refs.
IIUC, the tutorial is highlighing existing code, so replacing it should offer better readability with link to actual code, but it's still not going to be tested by cargo doc. (But if it links to an actual rust file that is compiled, so do we actually need it ?)

Exemple of how it looks:

icu4x/provider/core/src/hello_world.rs

Lines 105 to 122 in dbb02a1

	impl DataProvider<HelloWorldV1Marker> for HelloWorldProvider {
	fn load(&self, req: DataRequest) -> Result<DataResponse<HelloWorldV1Marker>, DataError> {
	#[allow(clippy::indexing_slicing)] // binary_search
	let data = Self::DATA
	.binary_search_by(|(k, _)| req.locale.strict_cmp(k.as_bytes()).reverse())
	.map(|i| Self::DATA[i].1)
	.map(|s| HelloWorldV1 {
	message: Cow::Borrowed(s),
	})
	.map_err(|_| DataErrorKind::MissingLocale.with_req(HelloWorldV1Marker::KEY, req))?;
	Ok(DataResponse {
	metadata: Default::default(),
	payload: Some(DataPayload::from_owned(data)),
	})
	}
	}

Only problem is that blob is linked to a specific version of the file, I don't think it can follow "head". (May be there's a trick I don't know about ?)
Updating github code ref should be more straightforward than changing code.

Let me know what you think :)

[Manishearth]

Only problem is that blob is linked to a specific version of the file, I don't think it can follow "head". (May be there's a trick I don't know about ?)

Replace the commit hash with main

[dsipasseuth]

I've tried, it doesn't work :'(

https://github.com/unicode-org/icu4x/blob/main/provider/core/src/hello_world.rs#L105-L122

[Manishearth]

that link works, it won't inline it though, if that's what you were wondering.

[dsipasseuth]

The point was to actually have it point to an actual code in repository, but displayable for doc.

Current doc right now points to files and have code snippet.
But files have changed a lot, and are definitely not up to date anymore.

I managed to point it back to where it moved.

[Manishearth]

Yeah that's not really possible, sadly.

[sffc]

I'm unfortunately moving this issue to the 1.1 milestone since it is not a strict blocker to the 1.0 code release. However, we still want the tutorials ready for the 1.0 blog post and publicity which is in the coming week.

[dsipasseuth]

I think we have a Chicken and egg problem in data_management.md. (mod version)

rust code need to compile, but using include on a non-existing mod.rs will fail the compilation.
So no binary to analyse in target folder.
Without binary, icu4-datagen will not run because --keys-for-bin is mandatory as far as I understand.

May be there's something I missed in the development process for the users ?

[Manishearth]

@dsipasseuth you can do mod data_management {} so you don't need to define a mod.rs

I don't quite understand the rest of your problem

[dsipasseuth]

It's not about the doc itself, it's about the tutorial itself.
I think the mod assume the application is already using ICU4X already. (either by blob or FS)

On an app that doesn't use ICU4X yet, and want to use it.

let's consider this app not using ICU4X

fn main {
  println!("Hello");
}

This commands fails. (there's nothing to generate, because... well, there's nothing !)

icu4x-datagen --cldr-tag latest --icuexport-tag latest --out my-data --format mod --keys-for-bin `target/debug/myapp` --locales ja

But if the user add this in his code, it doesn't compile anymore... because the file does not exists. (if it doesn't compile, there's nothing in target/debug/myapp to analyze for keys...)

include!("../my-data/mod.rs"); // defines BakedDataProvider

Not sure if I'm clear. May be it's just overthinking it.

[Manishearth]

Ah, I see, you're talking about the BakedDataProvider stuff.

So yes, you cannot use static data slicing if your application only supports BakedDataProvider. There are other ways to get the list of keys, and there are other kinds of providers you can use in the meantime.

[robertbastian]

I think the fix here could just be to not error in datagen if no keys are selected

[robertbastian]

Actually this is a proper bug in datagen, the Rust API allows no keys, so the CLI should as well. #2731

[dsipasseuth]

ok, doctest are being performed for tutorials whenever possible. closing this bug.