theme download gets stuck at 97% #801
Comments
|
Hey @alexissel Does it get stuck forever? Since v1.0 themekit, there are 2 things that would make it pause
If these things might be combined as well as the backoff in the retrying it might take about 10 seconds to finish or fail completely |
|
Also @alexissel, can you try running running in verbose mode ( |
|
Same issue here. Since the latest update I have been struggling to pull any themes. Now it seems to always stick at 97%. (Edit: it sometimes fails at other points too, however i've seen 97% more than once) -v shows it sticking at a standard .liquid file in the sections folder. runtime error: invalid memory address or nil pointer dereference Thanks |
|
Thanks, that's very helpful. It seems that when in non-verbose mode the error is being hidden so I've logged that as a separate issue #802. Are you able to determine which particular liquid files are not being uploaded? If so, can you check if you can upload any of them individually, e.g. |
|
Also, are you seeing a stacktrace below that error, e.g. with a reference to any |
|
See below for a couple of recent fails:
`[development] Successfully wrote sections/password-template.liquid to disk (Remote: b4a6f5b07911860fd00fd632cbfc1ea7, Local: )
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x145c8f2]
goroutine 300 [running]: `[development] Successfully wrote assets/swatch-get-fierce-leopard-print-coloured-insert-mug-green.png to disk (Remote: 2e0337daf78ca40484e5309f81c6e7e8, Local: ) goroutine 283 [running]: Realised that perhaps I'm wrong to say that it 'failed' on a certain file - the most I can specify is the last successful transfer. Also, after one of these fails, the rest of the download seem to 'pick up' where it left off by running the same theme get command. Cheers |
|
Thanks @andyw8 ! |
|
@andyw8 , until you'll release a fix for this, how can I downgrade themekit to v1.0.3 (latest release which works properly)? |
|
@alexissel You can run |
|
I have the same problem, I use themekit for many projects, but a couple of them get stuck at advanced download state. |
|
I am also having this issue with Themekit 1.1.0 and 1.1.0 there are no error when I run it in verbose mode, it just hangs between 97% and 99%. I have tried running the command multiple times to see if it could complete occasionally, but it seems on some themes it never completes. OS: Ubuntu 18 |
|
@alexissel We've fixed the issue and will have a new release of Theme Kit out very soon. |
|
@alexissel You can now run |
|
@andyw8 , the issue is still there... nothing changed on my end. |
|
@alexissel Thanks for reporting, we'll continue to look into. |
|
Also, still having an issue with this. No errors with verbose on, just stuck at 99% and two files remaining to download. I think it may actually be completing the whole theme download based on my git diff when I try to re-download with v1.0.3, but the visual representation of complete download or in verbose mode shows that it never finished and just hangs on v1.1.0, 1.1.1, and 1.1.2 |
|
Still having this issue as well on 1.1.2. tried downgrading to 1.0.3 and it seems to work. |
|
I have been looking into this and it only seems to happen on the first download of some themes. I just did it with one theme and it got stuck on a locales file for some reason Is it only happening on download for everyone? Notes for self
Update again: I added logging that looks like (and used Used some ruby results = {}
File.readlines('log.txt').each do |line|
next if line.strip == ""
parts = line.split(' ')
results[parts[0]] ||= {actions: [], fname: parts[2]}
results[parts[0]][:actions] << parts[1]
end
puts results.select { |id, res| res[:actions].count < 4 }And got 2 files that started downloading but never made it to the write stage. However the http adapter is not throwing response errors or throttling errors. My next steps are http client configuration, and retry mechanism might have a black hole in it somewhere still as well. Update Again Again: I am narrowing in on the http adapter. I believe since we are able to push through so many more requests, we are reaching the limits of the default golang http client. It now needs more configuration to handle idle connections more gracefully. |
|
Okay I have put up a draft PR with more information on where I think the issues lay. You can see it here #819 And you can try this patch out by running |
|
I have had this issue for a few times so far, here are my observations: @tanema, I tried your patch, unfortunately, didn't help. |
|
Can anyone else test this to see if it fixes it for you? I am fairly certain about where the problem lies, I just need to figure out the correct tweaks/settings. |
|
I have updated the |
|
Hey @tanema , I updated themekit to your version (v1.1.3-issue801) and I did the following:
So... I think it works?! |
This would mean just reverting to 1.0.2. There are some foundational changes since them and reverting them would just mean going back to that version number. If we knew the exact issues for this problem we would have fixed them of course.
That would not be the case if the changes were reverted
No, since developers are able to roll back themekit to an earlier version this means that they are not blocked from continuing their work. Without developers using this version and being able to identify which instances that it does work and which it does not, I have no recourse to fixing this issue and making themekit better. This change is actually working and improving development for a lot of developers but there is some edge case that we have not identified yet that is effecting a not-small sample of developers and we continue to work on this. |
|
Thanks, @tanema! I understand now. I thought there were two separate pieces of functionality that were added to last couple of versions:
If we could revert 1 and keep 2, that would be perfect. But seems like that's not possible...
I can send you the full theme file if you want. So far, it hangs 100% of the times. |
This has been implemented since version 0.0.1 I will be in contact to get theme information for you, that will be helpful, thank you. |
|
Hi, just wanted to comment that version 1.1.2 is still hanging for me at 99% on a basic Debut theme (hardly any customization as it is my dev store). I downgraded to 1.0.2 and that resolved it. |
|
Okay some things I have learned.
Is anyone having issue with any requests on themekit after their command pauses? For instance if you try to run theme download again, does it pause again? I ask because this looks like it is getting squashed by Shopify's bot detection. In trying to optimize themekit's functionality, we have made it a little bit too fast 😅 I may just simply have to limit the amount of inflight requests at a single time so that when a 429 response is received, another 20 requests are not still in flight causing our algorithms to keep counting requests. |
|
I noticed that if I run |
|
|
|
Has anybody been able to solve this? I've tried with v1.1.2 and v1.1.3-issue801 to download a theme with no luck. Update: v1.03 worked |
|
Yep, v1.0.3 is stable and is working perfectly. It doesn't make sense to update until this issue is fixed. |
|
I thought I'd comment here as I was having the same issue when using @shopify/themekit where the download would freeze at 99% when getting a theme from a store. I was using @shopify/themekit (v1.1.6) which was using themekit v1.1.1 as the binary under the hood. Downgrading @shopify/themekit to v1.1.4 fixed this for me as it's the most recent version to have themekit v1.0.3 which seems to be stable at the moment |
|
Same issue on 1.1.2. Got stuck at 174/175 - 99% Rolling back to 1.0.3 worked |
|
Okay I have opened a PR #839 to fix this. Thanks to @vfonic for lending me his theme to test. Please try the update again An unrelated note on the on the recent optimizations as well since I have seen the same confusion from people here. For the content checking (checksums) there has been some confusion on how that works. If you are only running download right now, you will not see it optimized. We were unable to backfill the (100s of) terrabytes of assets checksums so they will only be in effect if you run a deploy so that every asset will get updated with a checksum and you will see the optimization after that. Please let me know how the update goes. Since we are easing back on a lot of our internal work in preparation for BFCM I am hoping to ship a lot of my pending work in the next few weeks. |
|
Hey @tanema! Thanks for the time you put into this! I upgraded to I also tried couple of more times and I also setup a new Shopify store, uploaded the theme there and ran into the same issue. I've got couple of such errors: And then the very last line was: Another issue with this last line is that the program finished with exit code 0 (which means successful). Repro steps:
Expected:
Actual:
|
|
@vfonic by not finishing successfully do you mean finished with errors or hanging still?
There is an open PR to fix this.
These are not related. |
Thank you so much, you are absolutely right and this is very good news. I can address your other problems in separate issues. |
|
I agree with you. If you think that #839 PR fixes this issue good enough (my go skills are pretty bad) and that these other issues could be addressed in the following PRs, I think it's the best way forward, fixing one thing at a time. Thank you very much, Tim! |
|
Since we will not be merging anything around Black Friday/Cyber Monday, I have created a prerelease with the additions for this issue included in it. You can install the prerelease by running |
|
@tanema thank you for your efforts in resolving this, much appreciated. I will test and give feedback soon 👍 |
|
@tanema I upgraded from ThemeKit v1.1.1 by running I also tested other parts of the API (watching for file changes when developing, downloading a single file, listing themes and deploying) and everything seems to be working as expected. If anyone wants to test ThemeKit v1.1.3-pre using the node wrapper @shopify/themekit, I forked the project here so it automatically updates the binary to ThemeKit v1.1.3-pre after install. This could be useful for testing a pre release of the binary and testing CI/CD workflows where your project is dependent on using the node-wrapper instead of using the ThemeKit binary itself. OS: MacOS Mojave 10.14.5 |
|
I've run Test 1: Test 2: Test 3: Test 4: OS: MacOS Big Sur 11.0.1 |
|
Here's another test: [development] 353|353 [======================================================] 100 % [development] error downloading assets/mailing_list_bg.jpg: could not read response body [development] error downloading assets/banner_on_top_right.jpg: could not read response body [development] error downloading assets/block_collection_img.jpg: could not read response body |
|
@alexissel Thanks so much! The new error output is really helpful, though not that helpful because this tells me that the request succeeded but themekit could not read it for some reason. The request IDs showed me that the requests completed successfully on Shopify's side as well. This is helpful for #631 as it at least narrows down where the issues are stemming from. Thanks again for all the detail that you provided! Edit: By any chance, are those files really large? |
|
@tanema at least in my case I had issues with large files failing to download. Looking at the file names (images, settings_data.json), I'd assume these are large as well. BTW Thank you @tanema! I've been using themekit v1.1.3-pre and it works much better! I didn't know I could do |
|
@tanema I started my own investigation about the config/settings_data.json to find out why it doesn't download the file... |
|
v1.1.3 has been released with the fix in it. Please reopen this issue if the problem persists |
and others
Hi there,
I recently updated themekit to latest version (1.1.1) and since then it cannot download all the files...
I tried with 3 different themes and all of them get stuck around 97-98%
[development] Warning, this is the live theme on XXXXX-THEME.[development] 389|401 [===================================================================>--] 97 %Any thoughts on this?
Thanks
OS: Mac
Themekit: v1.1.1
The text was updated successfully, but these errors were encountered: