Feature: Migrate Downloaded Chapters

[fatotak]

Split from #1718

This is a QoL improvement feature for source migration of manga.

Currently, downloaded chapters cannot be migrated, and they must either be:

1. Deleted and redownloaded
2. Migrated manually via file system operations

There are problems with both approaches.

1. Delete + Redownload
   1. Can take a long time to download again.
   2. Unnecessarily erodes flash storage endurance.
   3. Unnecessary internet bandwidth consumption.
2. Migrate Manually
   1. Very time consuming for the user.
      1. User must download a sample of chapters to figure out the file naming pattern under the new source
      2. Rename the chapters according to the new source's chapter naming pattern
      3. Move the renamed chapters to the correct folder under the new source.
   2. Cache is out of sync with application until cache is invalidated.

This feature addresses the above issues with migrate.

Sample title for migration

Files in storage before migration

Migration interface (new checkbox for "Migrate downloaded")

Sample title post-migration

Files in storage post-migration

[AntsyLich]

I'm against this whole feature altogether. Downloads are not the same across sources. There are also edge cases here and there. Instead we should introduce some kind of merging feature alongside say move downloads to local

[fatotak]

I'm against this whole feature altogether. Downloads are not the same across sources. There are also edge cases here and there. Instead we should introduce some kind of merging feature alongside say move downloads to local

If you're against this feature, why aren't you asking for Chapter reader marker migration to also be removed? They share the same flaws. A purist approach would have neither feature to cater for the risk of mismatched chapters due to the edge cases you cite.

This feature is to make migrate actually useful for if/when the user is willing to take the risk.

[cuong-tran]

a few more points:

1. There is a lot cases where a manga having multiple chapters having same chapter number: different scanlators, different quality version, raw vs. translated, mis-match chapter number caused by volume number mixed in... While comparing old manga's chapter with new manga's chapter using chapter number might be the only way seems feasible, those cases I list will fail and overwrite downloaded files with each other.
2. While the chapter bookmark being migrated also has flaw like that but at least it's visible right there on the screen at a glance and easier for user to spot on, downloaded file will be hard for user to notice if it's really correct one or not unless they open & compare download files with online source (via webview?). A bit blindly maybe.
3. I also more of into the idea as Antsy (the maintainer) is moving the downloaded files to local source. Maybe still doing the rename to match new source's chapter name so user can decide to move the files into new manga's folder (at least the name is renamed for easier). Also, do a better handling of matching file name, for example if there is duplicated chapter number then keep old name instead of rename (to avoid overwrite each other), perhap?

[AntsyLich]

If you're against this feature, why aren't you asking for Chapter reader marker migration to also be removed? They share the same flaws.

No it doesn't. Chapter migration doesn't try to migrate stuff related to content like reading progress or history of a chapter

A purist approach would have neither feature to cater for the risk of mismatched chapters due to the edge cases you cite.

I hadn't even mentioned the edge cases you're just assuming. @cuong-tran has them in his point 1. And as you can none of them affects chapter migration in its current state

This feature is to make migrate actually useful for if/when the user is willing to take the risk.

So according to you the migrate feature at its current state is useless? Ok let's just assume what you said is true but if it comes with a huge maintenance burden (edge cases says hi) and there are alternative solutions which is simpler (i.e. entry merging) then I'm not gonna merge it even if it's a revolutionary feature.

[fatotak]

If you're against this feature, why aren't you asking for Chapter reader marker migration to also be removed? They share the same flaws.

No it doesn't. Chapter migration doesn't try to migrate stuff related to content like reading progress or history of a chapter

When I placed a read marker on the chapter (hold click chapter, click the tick box in the middle), and then migrated the manga, the migrated manga had that chapter as having been read. So what do you mean it doesn't come with the same flaws?

A purist approach would have neither feature to cater for the risk of mismatched chapters due to the edge cases you cite.

I hadn't even mentioned the edge cases you're just assuming. @cuong-tran has them in his point 1. And as you can none of them affects chapter migration in its current state

The existing function has the same flaws. (I.e you didn't read this version of the chapter from different scanlator or picture quality so on but it says you did)

This feature is to make migrate actually useful for if/when the user is willing to take the risk.

So according to you the migrate feature at its current state is useless? Ok let's just assume what you said is true but if it comes with a huge maintenance burden (edge cases says hi) and there are alternative solutions which is simpler (i.e. entry merging) then I'm not gonna merge it even if it's a revolutionary feature.

• Yes. Migrate in its current state only saves a few seconds for the user, but is followed by activity taking hours or days (redownload). Without chapter file migration, it's pointless. Consider the case where there is 250gb from 1 source. When the source dies and need to migrate... It takes 8weeks+ to download, and kills sd card if need to migrate multiple times.
• The problem gets even worse as post migration, can't use download all button as it only downloads unread, so need to click dl button 1 by 1 or clear read marker, download all, then reapply read marker.
• Look at the alternative (manual) method of migration compared to the current migrate function:

Step	Current migrate	Manual migrate
1	click migrate	click title
2		click source
3		click search
4		paste
5	select applicable entry	same
6	select checkboxes	click favourite
7	click migrate
8	hold click chapter
9	select all
10	clear read progress
11	click download all	same
12	hold click last read chapter	same
13	click mark all as read (tick with down arrow)	same
14		back to title of old source
15		click unfavourite
16		click delete

When comparing these 2 workflows, the current migrate only saves a total of 2 clicks while suffering the SAME drawback as this new function. This new function saves a few clicks from above, but more notable savings are:

If re-downloading all

1. long download time
2. sd card wear

If trying to migrate manually

1. Very tedious manual process taking anywhere from 30s to 5 mins per title to rename and move the files correctly <<< this manual process is what's implemented in this PR, in a more efficient manner (direct access to manga meta data, rather than figuring out the meta data manually by downloading and cancelling a sub-sample of the chapters when the tmp folder is created by the downloader) by this function.

I already acknowledged that it won't work for certain edge cases, but the existing function also doesn't work for the same edge cases.

What alternative solutions? What entry merging? Can you provide link or detailed instructions so I can try? If there is an alternative solution that will solve my problem I'll be happy to use it. The problem being solved are:

1. Avoid long download time before resuming to read.
2. As the time between reading chapters from same title may be long (months, or even years, especially for irregular or infrequent update titles) and I can't understand the content sometimes, need to traverse back many chapters. So need to keep all the read marked chapters to accommodate this scenario (i.e. click resume, can't understand, unmark last n chapters from having been read, click resume again, repeat until remember the content).
3. The above problem gets even worse (time between reading chapters from same title) as I don't like reading 1 chapter at a time and will wait to accrue n chapters (minimum 5) to read before resuming to read.
4. Avoid/reduce sd card write wear and killing of sd card!!! If DMCA go on a rampage, you migrate from one source to another... after finishing many of the download, the new source gets taken down... I remember last time tachiyomi killed my sd card was when there was such a crackdown, and had to redownload 3 - 4 times x 250gb.

[AntsyLich]

When I placed a read marker on the chapter, and then migrated the manga, the migrated manga had that chapter as having been read. So what do you mean it doesn't come with the same flaws?

"related to content" I have read chapter 1 is different from I have read chapter from this source. I repeat we don't migrate reading progress or history of a chapter which is related to content.

The existing function has the same flaws. (I.e you didn't read this version of the chapter from different scanlator or picture quality so on but it says you did)

Same as above

Yes. Migrate in its current state only saves a few seconds for the user, but is followed by activity taking hours or days (redownload). Without chapter file migration, it's pointless. Consider the case where there is 250gb from 1 source. When the source dies and need to migrate... It takes 8weeks+ to download, and kills sd card if need to migrate multiple times.

The problem gets even worse as post migration, can't use download all button as it only downloads unread, so need to click dl button 1 by 1 or clear read marker, download all, then reapply read marker.

Look at the alternative (manual) method of migration compared to the current migrate function:

So the thing is migration is useless for you in particular. Most of the user base are not data hoarders so there usually isn't a demand/request for download migration in this scale.

What alternative solutions? What entry merging? Can you provide link? If there is an alternative solution that will solve my problem I'll be happy to use it.

Quoting myself from the first comment

Instead we should introduce some kind of merging feature alongside say move downloads to local

Look I'm not saying the problem is non-existent but we should be looking at this from a boarder view. The solution I'm proposing is a very requested feature we could be killing two birds with one stone this way.

[fatotak]

"related to content" I have read chapter 1 is different from I have read chapter from this source. I repeat we don't migrate reading progress or history of a chapter which is related to content.

??? As a user, all I see is "did I read this" chapter? yes/no.
I don't know what you mean by reading progress or history of a chapter if what you are talking about is different to the read marker that I'm referring to (i.e. hold chapter, click tickbox byitself|with down arrow|with slash).
When I migrate, the "did I read this"/"read marker" is set, even if the content is different.

Quoting myself from the first comment

Instead we should introduce some kind of merging feature alongside say move downloads to local

Look I'm not saying the problem is non-existent but we should be looking at this from a boarder view. The solution I'm proposing is a very requested feature we could be killing two birds with one stone this way.

So the solution doesn't exist yet?
Why not take this one up temporarily, and then delete it once the permanent one is available?
Put warnings on the checkbox if you need to, so it's 100% at the user's risk.

[AntsyLich]

"related to content" I have read chapter 1 is different from I have read chapter from this source. I repeat we don't migrate reading progress or history of a chapter which is related to content.

??? As a user, all I see is "did I read this" chapter? yes/no. I don't know what you mean by reading progress or history of a chapter if what you are talking about is different to the read marker that I'm referring to (i.e. hold chapter, click tickbox byitself|with down arrow|with slash). When I migrate, the "did I read this"/"read marker" is set, even if the content is different.

You do realize we have a history tab right?

Quoting myself from the first comment

Instead we should introduce some kind of merging feature alongside say move downloads to local

Look I'm not saying the problem is non-existent but we should be looking at this from a boarder view. The solution I'm proposing is a very requested feature we could be killing two birds with one stone this way.

So the solution doesn't exist yet? Why not take this one up temporarily, and then delete it once the permanent one is available? Put warnings on the checkbox if you need to, so it's 100% at the user's risk.

And who is going to take the maintenance burden of this temporary feature?

[fatotak]

You do realize we have a history tab right?

Yes... what about it?
How is the history tab related to what we were talking about (accuracy of the chapter when comparing old source vs new)?
This change doesn't touch anything in the history tab, just moves files from old folder to new under the different naming conventions across sources.
Anything it fails to match, it will leave orphaned in the old source folder (subsequently deleted by the delete downloaded migration option).

And who is going to take the maintenance burden of this temporary feature?

I'll commit to maintain it and then delete it in 3 months time, giving you guys 3 months to figure out and implement the permanent solution. Is that an ok resolution?

I've an immediate need (due to a couple of sources changing into a new source) affecting over 250 titles. It took me 2+ weeks (~ 4 active interaction hours) to migrate 50 titles manually due to different naming standards between the old source and new. If I continue the manual method, it will take who knows how long with 20+ interaction hours needed.

This change together with my other one will also ensure my SD card won't die prematurely, which costs $100+ per card (I have a large one that fits my massive downloaded library). Given my SD card usage is only: Tachiyomi (total ~400Gb), DCIM (< 30 pics / yr = ~ 100Mb) and Google Maps data (~300Mb), any premature death of the SD card from write wear will always be Tachiyomi (bad download method in respect to flash write wear, need to redownload for dead sources).

You can also do a detailed write up of your proposed permanent solution (raise an issue with the solution description and link the issue here so I can find it). If I think it's something I can help with to fast track resolution for my problem, maybe that's another approach...
But a brief 1 liner isn't something I can understand fully, so I have no confidence to help with.

[krysanify]

+1 to have a "merging to local source" instead. there are a few times when I downloaded chapters from one source, but need to download some missing chapters from a different source.

it would be a shame to tied that usefulness to a migration process since i don't use that second source very often.

[Jetspectre]

The problem with this approach is that not all sources are the same. Source A could host 20 pages per chapter, but Source B has the SAME CONTENT but cut differently increasing the page numbers to 50 or even 100+. This much is TRUE for every Korean Manhwa that is available (Scanlated vs Official).

Chapter Reader marker migration is a different issue, because the core content of each chapter is consistent across sources. The main difference lies in the inclusion of scanlator pages, credits, or explanatory notes, which vary between sources but don't affect the core chapter content.

While this may work for you, it most certainly will not work for up to 9X% of users.

Some recommendations:

1. Migrate from Source A to Source B -> Delete orphaned downloads -> Download new ones
2. Host a local server with Komga/Mango/Kavita
3. Buy a new phone where you don't have to rely on SD cards, or buy a new OTG thumb drive
4. Make a personal fork with this feature implemented

[krysanify]

Tachiyomi (total ~400Gb)

The more you explained why this feature is needed, the more niche I find the user demographics to be and the furthest away for it to be potentially useful for users that use mihon just to read and not hoarding data.

You can say, "so what? it's an opt-in.", but it's not clear why this needs to be in main project of mihon other than your personal needs.

If I may make a suggestion: find a way for this to work on your fork, share it with the community and see how many opted in. Google Play Store policy requires dev to test their apps on at least 12 beta testers, that could be a good amount to aim even though we're not in the play store.

[fatotak]

The problem with this approach is that not all sources are the same. Source A could host 20 pages per chapter, but Source B has the SAME CONTENT but cut differently increasing the page numbers to 50 or even 100+. This much is TRUE for every Korean Manhwa that is available (Scanlated vs Official).

You think most people actually care that pages are cut differently?
Ridiculous.

Chapter Reader marker migration is a different issue, because the core content of each chapter is consistent across sources. The main difference lies in the inclusion of scanlator pages, credits, or explanatory notes, which vary between sources but don't affect the core chapter content.

Nope. The cases where it actually matters, they are both broken. For all other cases, it's fine.

• Qualitative differences (resolution, translation, etc) - both broken, but suspect minor issue (most people don't care?).
• Censorship - both broken (suspect people care about this one)
• Chapter numbering differences (source A starts at 0, source B starts at 1, source A breaks chapter into 1.1, 1.2, 1.3, source B doesn't break chapter, etc) - both broken, and I think this one is the most critical one.

Given the most critical one is broken on both, I'm not sure why people are getting their panties in a knot for the more minor cases.

While this may work for you, it most certainly will not work for up to 9X% of users.

I think that's a ridiculous assumption.
It's been acknowledged that there certainly are cases where it won't work, but under the SAME cases, the existing chapter read marker also doesn't work.

1. Migrate from Source A to Source B -> Delete orphaned downloads -> Download new ones

Sigh... this change is to exactly AVOID this.
No matter what you think, current migrate is pretty useless, as it saves a total of 2 clicks (same can be achieved in other ways with 2 more clicks than migrate). If I cared about accuracy (i.e. the 3 issues outlined above), I wouldn't bother using migrate in the first place. Where I care about convenience more than accuracy, then migrating already downloaded chapters is the biggest convenience (saves hours vs manually)

3. Buy a new phone where you don't have to rely on SD cards, or buy a new OTG thumb drive

And kill your $2000 phone instead of your $100 sd card?
You're such a genius.
Read up on flash wear, and read up on what kind of storage phones, sd cards and thumb drives use (hint: they are ALL flash)
Actually, don't bother.
You just keep downloading on your internal storage and kill your phone.

[fatotak]

The more you explained why this feature is needed, the more niche I find the user demographics to be and the furthest away for it to be potentially useful for users that use mihon just to read and not hoarding data.

You are assuming everyone except me uses it like you do.
That's false by default.

Minimally, there are already 3 people who want/need this feature (100% of the people I know in real life who use this app - 2 others + me), which is why I decided to write the code and raise this PR in the first place. But most users are actually silent... they don't participate here (or any other comms channels) and just expect devs to magically read their minds and eventually implement the features they need. So while you may think I'm a unicorn, I don't believe I am, based on my small sample data (3 ppl, 100% hit rate).

You can say, "so what? it's an opt-in.", but it's not clear why this needs to be in main project of mihon other than your personal needs.

This feature will be:

1. Useful for people like me
   • Wifi is unmetered, mobile data is metered, and don't want to pay extra for data
   • Don't particularly care about scan quality, translation quality, etc - just want to read
   • Tendency to read in bulk, rather than one chapter at a time (minimum 5 chapters, preference to read 1 entire arc at a time)
   • Tendency to forget some of the content and read the previous chapters.
   • Tendency to glance at an old chapter when referenced (i.e. chap 452 might have a returning character and a foot note saying "see chap 137", triggering to re-read chap 137 - 142 for a refresher for example).
   • Don't delete chapters until a series is finished and then delete the entire series once the last page of the series has been read.
2. Especially useful for people unlike me
   • People who have to pay for data (i.e. have metered wifi such as wireless broadband, OR only have mobile data)
   • Do keep some chapters on their device
3. Useless to people who
   • Delete chapters after reading
   • Only read chapters once
   • Do care about differences in translation quality, scan quality, etc
   • Are happy to use mobile data rather than using wifi only

If I may make a suggestion: find a way for this to work on your fork, share it with the community and see how many opted in. Google Play Store policy requires dev to test their apps on at least 12 beta testers, that could be a good amount to aim even though we're not in the play store.

If I release my fork and take up the maintenance burden, why would I need to come back to mainline?
I'm trying to avoid releasing my own fork as it does come with a maintenance burden (all the other stuff apart from the few features I specifically need added - you'll see that both my changes are about 1 thing - reduce flash write erosion, paying a small price to achieve this. In my other change, the price was RAM. In this change, the price is the potential that the content is slightly different [txn quality, scan quality, etc]).

I will be happy to support my changes in mainline for a period of time to achieve not having to maintain the rest of the project in my own fork.

[taos15]

The problem with this approach is that not all sources are the same. Source A could host 20 pages per chapter, but Source B has the SAME CONTENT but cut differently increasing the page numbers to 50 or even 100+. This much is TRUE for every Korean Manhwa that is available (Scanlated vs Official).

Chapter Reader marker migration is a different issue, because the core content of each chapter is consistent across sources. The main difference lies in the inclusion of scanlator pages, credits, or explanatory notes, which vary between sources but don't affect the core chapter content.

While this may work for you, it most certainly will not work for up to 9X% of users.

Some recommendations:

1. Migrate from Source A to Source B -> Delete orphaned downloads -> Download new ones

2. Host a local server with Komga/Mango/Kavita

3. Buy a new phone where you don't have to rely on SD cards, or buy a new OTG thumb drive

4. Make a personal fork with this feature implemented

One thing people keep forgetting is that most source just re-uploaded (aggregators) chapters from other sources. So the migration will do the same as any source do. For example, there are many cases that the first version of a chapter get updated, if you dont delete and re-download the chapter you would never know.

I think this is a very useful features. It also helps save bandwidth, and lower the chance a source trying to break working with miho due to abuse. Imagine having a manga that is 3000+ chapters, and the source that you were using is not available anymore, why re-download everything?

The numbering should not matter, there are evern some source that release .5 version of old chapters and we just mark them as read, so I dont see why should we be able to add any chapter that we already have.

[taos15]

I'm against this whole feature altogether. Downloads are not the same across sources. There are also edge cases here and there. Instead we should introduce some kind of merging feature alongside say move downloads to local

Actually this sounds like a good idea. A local or even a new default library like merged, where you can just get all that match name:xyz. That library will get the manga that match that name, and show all your downloaded chapters. This will need a source of true for it to work correctly, something like only showing manga that have a tracker, that way the tracker ID can be use to identify manga that are the same (since the naming, and the id of the source might vary).

[fatotak]

local or even a new default library like merged, where you can just get all that match name:xyz.

Although that may sound like a good idea, the problem is that it will be a very big change to how Tachiyomi works, as well as complicated to implement.

1. The base model of Tachiyomi has Source as the top of the food chain. The change that you guys are proposing fundamentally changes how the model works, with Title being the top of the hierarchy, and sources and chapters being subsidiaries.
2. Then design decisions need to be made about how chapter downloads will work... more specifically, will it download the same chapter from all sources? Or will it selectively download? If selectively downloading, how will it choose the source to download from when there are multiple sources available with the same chapter?

I believe the feature that is implemented in this PR, although not useful for every Tachiyomi user, is still useful for many users.
I have been unable to use the app in the last few months due to the closure of some sources and the annoyance of migrating titles manually, prompting me to work on this feature.

You can talk about ideals, but they don't exist yet, and even if someone starts designing it now, I really don't see it happening for the next 12 months to be honest, given all the regression impact it will have, so I don't see why people are against this simple solution, even as a temporary stop-gap measure.

[taos15]

local or even a new default library like merged, where you can just get all that match name:xyz.

Although that may sound like a good idea, the problem is that it will be a very big change to how Tachiyomi works, as well as complicated to implement.

I understand you point, I was not suggesting a rewrite of how the current app work, I was think kore like a second local which I think there are very few that actually use it. If am not wrong --I have comtributed to suwoyomibut never mihon, but there are based on the same code based-- the current logic is based on source, but dont we use source's ID? What I was proposing was not changing that logic, but instead reused that implementation to treat trackers ID like a source "only for that new library".

so the tracker will act like a source, instead of getting manga with ID 1234 from source with ID 678; that library will be pulling manga ID 2222 from tracker ID 5555.

It will not affect any other library, and should onmy display manga that have a tracker.

I try to comment in discord, but I think I domt have access to the programming channel.