When pushing to Central, push form definitions for all versions referenced in submissions

[lognaturel]

Central only accepts submissions for form versions that it is aware of. Aggregate allows limited form versioning but only retains the latest form version. Briefcase only has awareness of the last form version it pulled.

There was a v2 planned to address this and other limitations but unfortunately we've been struggling to get the time and resources to make it happen.

TODO:

• Canceling push should not give Success status
• Investigate key issue -- is it from pulling new form definition version?
• When pulling Kasia's original All Widgets form with v1.17.4, are there blank submission directories? Seems this was related to the server running out of memory
• Attempt to push submission attachments should not result in 404 - Seems this was also related to server running out of memory
• Look into progress bar starting after warning dialog dismissed
• Seems cancel button on warning dialog is totally broken
• Look into empty/missing form versions. May need updates to Central API to handle fully. Possibly file as separate issue once merged. - Support empty form versions when doing multi-version push to Central #891
• Write docs page - Document using Briefcase with Central docs#1272
• File issue for making docs link in warning dialog clickable - Make documentation link in Central push warning dialog clickable #890
• In 1.17.4, pull skips media attachments if the submission exists. That doesn't seem right. It means it can be really hard to know a dataset is incomplete.

This PR:

• reads versions from submissions as it pulls them from Aggregate, Central, or a Collect directory and stores a set of all identified versions in the form metadata
• when pushing to Central from the UI, warns the user that the same form definition will be pushed to represent all the versions needed by submissions
• when pushing to Central, for each identified version from submissions, pushes the available form definition as a draft and then publishes it with the version identifier
• when pushing to Central, pushes and publishes the latest form definition last
• only uses formId in metadata lookup instead of formId and version. The implication of using the formId, version pair was that metadata would be lost each time a new version was pulled (e.g. export status or submission versions)
• does not add any kind of warning for the command line push to Central. It's a less common interface and its users are more savvy.

Acceptance criteria:

• Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from this version of Briefcase
  And: the submissions are all for the active form definition's version
  When: I push that form and submissions to Central
  Then: the form definition and corresponding submissions are pushed without any special behavior

• Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from this version of Briefcase
  And: some submissions are for the active form definition's version and others for prior form versions
  When: I push that form and submissions to Central
  Then: I see a warning with a description of what will happen, a reference to a documentation page
  And: I see buttons to Cancel or Push

• Given: I see a warning about pushing a form with submissions for multiple versions to Central
  When: I click the Cancel button
  Then: the push is aborted and nothing is pushed to Central

• Given: I see a warning about pushing a form with submissions for multiple versions to Central
  When: I click the Push button
  Then: the push begins

• Given: I have confirmed a push to Central
  And: there was no form with the corresponding formId on my Central server
  Then: all form versions referred to by submissions are present on my Central server
  And: all submissions are pushed
  And: the form version Briefcase knows about is the actively published version

• Given: I have confirmed a push to Central
  And: there was a form with the corresponding formId on my Central server
  And: some versions that Briefcase has submissions for were not previously uploaded to Central
  Then: all form versions that were not previously uploaded to Central are now on my Central server
  And: all submissions are pushed

• Given: I have confirmed a push to Central
  And: there was a form with the corresponding formId on my Central server
  And: all versions that Briefcase has submissions for were previously uploaded to Central
  And: the form was deleted
  Then: the form is in draft state on my Central server and can't be published because of a version conflict
  And: I see errors about all of the form versions and submissions failing to upload

• Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from a prior version of Briefcase (<=v1.17.4)
  And: some submissions are for the active form definition's version and others for prior form versions
  When: I push that form and submissions to Central
  Then: submissions for versions other than the version Briefcase knows about fail
  And: in the details, I see a message about trying again and a reference to documentation

• Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from a prior version of Briefcase (<=v1.17.4)
  And: some submissions are for the active form definition's version and others for prior form versions
  And: I let a push attempt complete and fail
  When: I retry
  Then: all form versions that were not previously uploaded to Central are now on my Central server
  And: all submissions are pushed
  And: the form version Briefcase knows about is not the actively published version

• Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from this version of Briefcase
  And: some submissions are for the active form definition's version and others for prior form versions
  When: I push that form and submissions to Central from the command like
  Then: all form versions that were not previously uploaded to Central are now on my Central server
  And: all submissions are pushed

[ggalmazor]

Just saw this :) I'm not sure if it's ready, but I will take a lookie look anyway and see how it goes ;)

[ggalmazor]

In released versions, this means that if a user exports submissions and then pulls a new form version, exports "from last" will start at the beginning because the metadata about last exported submission will be lost. @ggalmazor maybe you have some insights.

Oops. That sounds like an unintended behavior. That information shouldn't be lost between pulls

[ggalmazor]

@lognaturel, I did a quick review and I think your approach is OK. Some thoughts/notes (probably things that you already know, though):

• Use UncheckedFiles where possible to deal with the generation of the form.
• Remove duplication between blocks that push generated form versions and the local form

[lognaturel]

Thanks so much for taking a look, @ggalmazor, and for all the helpful feedback.

That sounds like an unintended behavior. That information shouldn't be lost between pulls

Interesting! Do you happen to remember why version is included in FormKey? I expected the form metadata repository to be keyed on logical forms rather than specific versions. Is there a reason not to do that?

[lognaturel]

@getodk/testers There's a lot of behavior here and weird edge cases depending on what versions are already on the Central server and when forms were pulled. I think I've come up with something that makes sense and handles most of them (except for forms with blank versions) but before we do code review it would be helpful for you to do a discovery pass so we identify anything missing or that just doesn't make sense.

[kkrawczyk123]

@lognaturel I have another dose of questions/concerns:

Given: I have confirmed a push to Central
And: there was a form with the corresponding formId on my Central server
And: all versions that Briefcase has submissions for were previously uploaded to Central
And: the form was deleted
Then: the form is in draft state on my Central server and can't be published because of a version conflict
And: I see errors about all of the form versions and submissions failing to upload

Briefcase's status is "Success with errors" what is acceptable as draft form version is pushed but I can see another issue here which is actually more connected to Central than to Briefcase. I went to project and tried to publish draft version and I can't to that, text field to specify version is missing what makes form useless right now.

Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from a prior version of Briefcase (<=v1.17.4)
And: some submissions are for the active form definition's version and others for prior form versions
When: I push that form and submissions to Central
Then: submissions for versions other than the version Briefcase knows about fail
And: in the details, I see a message about trying again and a reference to documentation

I can't see the difference between case where I am pulling form with submissions of different form versions using Briefcace 1.17.4 or PR-version and then pushing it again to new project on Central using PR-version: all submissions are sent and errors about failing to push media files are displayed only.

Given: I have pulled a form definition and submissions from Aggregate, Central, or a Collect directory from a prior version of Briefcase (<=v1.17.4)
And: some submissions are for the active form definition's version and others for prior form versions
And: I let a push attempt complete and fail
When: I retry
Then: all form versions that were not previously uploaded to Central are now on my Central server
And: all submissions are pushed
And: the form version Briefcase knows about is not the actively published version

Here as a step "I let a push attempt complete and fail" you mean use Briefcase 1.17.4 in first push and the second push using PR-version? I do not fully understand the difference between that scenario and the other above. I have the same results.

I am wondering how the last published version is actually selected on Central? Does it happen randomly? (ex. project 193). See the gif:

[lognaturel]

errors about failing to push media files

I'm very puzzled about what's going on here. Do you have a form with a single version and media files that you could do a pull and then push with 1.17.4 with? I suspect there's an existing issue.

I went to project and tried to publish draft version and I can't to that, text field to specify version is missing what makes form useless right now.

I think you got in this state by deleting a form with that form version and then trying to create it again. I agree that this can lead to a user being stuck and will explore with the Central team.

I can't see the difference between case where I am pulling form with submissions of different form versions using Briefcace 1.17.4 or PR-version and then pushing it again to new project on Central using PR-version

You'd have to make sure to clear your Briefcase directory before doing a pull from <=1.17.4 so that the form version cache is cleared. Then also make sure you're pushing to a project that doesn't know about that form at all.

you mean use Briefcase 1.17.4 in first push and the second push using PR-version?

No, the first push would have to be using the PR version. So this is download everything with Briefcase 1.17.4 after having cleared the Briefcase directory. Then go to the PR version and push. There should be failures. Then try again and it should succeed. However, the published version will not be the latest one. This I will describe in documentation. As you noticed:

I am wondering how the last published version is actually selected on Central? Does it happen randomly?

If you already had things pulled in <=1.17.4, there's no information about submission versions. That means Briefcase will only push the form version it knows about and will publish it. When you push again, it will know about older form versions and push those in an unpredictable order. Which ever one is last will be published. So yes, the published version is entirely unpredictable in that case. In the case where the pull happened with this PR, the order of the versions before the published one is also unpredictable. EDIT: I now push obsolete versions in alphanumeric order so at least coming from Aggregate the order will generally be sensible.

[kkrawczyk123]

• I was thinking and trying things in empty submission context. If by "empty submission" you mean one that they don't have submissionDate in exported by Briefcase file. I think that all submissions filled and send to Central directly do not have that in csv created by Briefcase and it was like that before and it differs from submissions sent to aggregate and then pushed to central. In Downloaded csv from Central all submissions have submissionDate probably exporting mechanisms are different?
  The second suspicious thing for me is that: Form project 197 I pulled all widgets with 1.17.4 I can see that all 256 submissions are pulled to my sd but just theoretically some of instances folders are empty. When I used the latest pr-version I have noticed that only 226 are pushed so I started to investigate. Maybe this is the empty submission issue case. The form has submissions of different versions that 1.17.4 does not recognize. But those submissions are not visible in exported file.

• I have encountered on weird thing while trying to reporoduce issue with failing media. I added form manually to central and send submissions with media attachments from Collect project: https://test.central.getodk.org/#/projects/198 I can pull it using briefcase but I can't push it to another project where that form was not previously added. What I don't understand. When I tried the same push via cmd I had "Unexpected error in Briefcase. Please review briefcase.log for more information. For help, post to https://forum.getodk.org/c/support" the file: All widgets.zip
  error from log file:
  2020-10-14 12:07:18,586 [main] ERROR org.opendatakit.briefcase.Launcher - Error java.lang.IllegalStateException: Duplicate key FormKey{name='All widgets', id='all-widgets', (version=Optional[202007150])} (attempted merging values FormMetadata{key=FormKey{name='All widgets', id='all-widgets', (version=Optional[2017121301])}, storageRoot=/home/kasia/Desktop/ODK Briefcase Storage, formDir=forms/All widgets/instances/uuidc1f7a209-5b31-4570-ad61-2ecc72cacac7, hasBeenPulled=true, cursor=EmptyCursor{}, lastExportedSubmission=Optional.empty} and FormMetadata{key=FormKey{name='All widgets', id='all-widgets', (version=Optional[202007150])}, storageRoot=/home/kasia/Desktop/ODK Briefcase Storage, formDir=forms/All widgets, hasBeenPulled=true, cursor=EmptyCursor{}, lastExportedSubmission=Optional.empty}) at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:133) at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180) at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) at java.base/java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:442) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.syncWithFilesAt(FileSystemFormMetadataAdapter.java:78) at org.opendatakit.briefcase.model.form.FileSystemFormMetadataAdapter.at(FileSystemFormMetadataAdapter.java:36) at org.opendatakit.briefcase.push.central.PushToCentral.push(PushToCentral.java:98) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.opendatakit.briefcase.reused.job.JobsRunner.launchAsync(JobsRunner.java:63) at org.opendatakit.briefcase.operations.PushFormToCentral.pushToCentral(PushFormToCentral.java:100) at org.opendatakit.common.cli.Cli.lambda$run$4(Cli.java:130) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.opendatakit.common.cli.Cli.run(Cli.java:127) at org.opendatakit.briefcase.Launcher.main(Launcher.java:90) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
  I was able to push form only when I deleted submission.

I'm very puzzled about what's going on here. Do you have a form with a single version and media files that you could do a pull and then push with 1.17.4 with? I suspect there's an existing issue.

• I've tried to reproduce it somehow and I have literally nothing. All media added to submissions are pushed when I use only one form version, different kinds of files. I have observed a potential cause in the media folder created after export data that I was using before: some of media do not have file type (I was not manipulating with them manually) so there is a chance that's the reason why they can't be pushed. There might not be any issues with normal media files.
  

• I was confused by fact that Push dialog did not appear for my always even after clearing prefs. Finally, I was able to reproduce that, see the gif. I am switching Briefcase versions but use the same sd, clearing prefs between each run, central project that I have used is newly created and empty before the push. Push result under those conditions is 37 submissions from 256 what is a little weird too as push is done on the newest pr-version but result looks more like from 1.17.4. When I change sd in the same scenario dialog displays.
  

• When I interrupt pushing using Cancel button - the operation is canceled but success status is displayed, see the gif:
  
  Instead of display that push has been canceled, result from the same scenario from 1.17.4:
  

[lognaturel]

Thanks for all the detailed checks, @kkrawczyk123. Before you dive back in, I encourage you to read through getodk/docs#1272. Hopefully that will clarify the intended behavior for weird edge cases. If not, let me know what parts are unclear!

by "empty submission" you mean one that they don't have submissionDate in exported by Briefcase file

I was seeing instance folders with absolutely no contents, not even submission.xml. It turns out to be related to the export issue you were having -- the QA server just didn't have enough memory to deal with the largest media files. I've now added swap and everything is behaving as I would expect. I'll work on updating Central docs so this is really clear. I didn't expect these issues with relatively small media files and I forgot we didn't have swap on that machine. If there's anything else expected going on related to transfer, memory should be our first suspicion. I think you're right that this is what you experienced when you pulled 256 and ended up pushing 226.

I'm not seeing any failures to push media files anymore so maybe that was also related?

use the same sd

The form versions that need to be created on Central are identified on pull. If you pull on v1.17.4 and use the same storage directory to push using this PR, older form versions won't be pushed because the form version list is stored in the storage directory and v1.17.4 just doesn't record that info. But the process of pushing on this PR generates the list of form versions to push. So once you've attempted a push, that storage directory knows about all the versions, and a future push with this PR will send all form versions. Does that make sense? This might be one to hop on a quick call about if you think there might be a bug or if you're still not feeling clear on the intended behavior (don't worry, it's super hard to reason about).

When I interrupt pushing using Cancel button - the operation is canceled but success status is displayed, see the gif:

Is it possible this only happens if you cancel right after the "Start pushing form definition..." message is emitted but before anything actually gets pushed? I can't get that behavior. If I cancel immediately right after tapping push, the cancel has no effect. This is a bad state but it's the same on v1.17.4. It's also unlikely so I'm not too worried about it. If I push once submissions are sending, I see something like

Submission 10 of 1902 sent
Sending submission 17 of 1902
Submission 11 of 1902 sent
Sending submission 18 of 1902
Submission 12 of 1902 sent
Sending submission 19 of 1902
Submission 13 of 1902 sent
Sending submission 20 of 1902
Cancelled by user
Submission 14 of 1902 sent

In that case, it's annoying because the "Cancelled" message may not be the last one. But I think it's ok.

The most similar case I can get is if I cancel right after form creation:

Start pushing form and submissions
Creating form (all-widgets)
Cancelled by user
Form created (all-widgets)
Operation cancelled - Push form version "2017053001"
Operation cancelled - Publish form version "2017053001"
Operation cancelled - Push form version "2017053003"
Operation cancelled - Publish form version "2017053003"
Operation cancelled - Push form version "2017121301"
Operation cancelled - Publish form version "2017121301"
Operation cancelled - Push form version "2017121302"
Operation cancelled - Publish form version "2017121302"
Operation cancelled - Push form version "2017121304"
Operation cancelled - Publish form version "2017121304"
Operation cancelled - Push form version "2017121307"
Operation cancelled - Publish form version "2017121307"
Operation cancelled - Push form version "2018022001"
Operation cancelled - Publish form version "2018022001"
Operation cancelled - Push form version "2018051401"
Operation cancelled - Publish form version "2018051401"
Operation cancelled - Push form version "2018051409"
Operation cancelled - Publish form version "2018051409"
Operation cancelled - Push form version "2018081601"
Operation cancelled - Publish form version "2018081601"
Operation cancelled - Push form version "2019020201"
Operation cancelled - Publish form version "2019020201"
Operation cancelled - Push form version "2019020208"
Operation cancelled - Publish form version "2019020208"
Operation cancelled - Push form version "2019051302"
Operation cancelled - Publish form version "2019051302"
Operation cancelled - Push form version "2017121308"

In that case it looks like there's no final message.

I have encountered on weird thing while trying to reporoduce issue with failing media. I added form manually to central and send submissions with media attachments from Collect project: test.central.getodk.org/#/projects/198 I can pull it using briefcase but I can't push it to another project where that form was not previously added.

Wow, that was indeed crazy! The problem was that one of the submissions had an XML media attachment. Briefcase was treating that like a form definition and then getting confused because there were two form definitions at different paths. Should be fixed, now.

I believe I have addressed all of the issues you highlighted except that I wasn't able to satisfactorily reproduce the "Success" after cancelling issue. Does that sound right? If not, please summarize issues from above that are still present so we make sure we're on the same page. 🙏

[mmarciniak90]

@lognaturel I noticed a different behavior in one case. Probably it's uncommon.

1. I pulled 3 different versions of form via Form definition - form 1648 from odk.zip
2. Then I tried to push it to Central

The form was pushed with errors on v1.17.4. It appeared in Central.

But when I tried this case with these changes, I clicked on the push but nothing happened.

[kkrawczyk123]

List of test cases verified/retested so far:

• progress bar issue has been fixed
• cancel button behavior on push information dialog has been fixed
• push to Central form with submissions of different versions
• push to Central form pulled from Aggregate
• push to Central form pulled from Central
• push to Central form with submissions of different versions and media
• push form with media and submissions with media attached
• push form with submissions of different versions when there is already draft form on Central
• push form with submissions of different versions when there is already published form on Central
• abandon draft, push again
• add another version of form, push again
• pull and push submissions filled on Enketo
• delete a form, push again: already mentioned in comments above (cannot publish form on central)
• able to download pushed results from central
• encrypted form on Central, push form with submissions of different versions: pushed form versions and submissions are unencrypted, already mentioned
• pull from odk folder, push to Central
• form without a version specified: only draft version is pushed
• form pulled with 1.17.4 and pushed: only form known by briefcase and it's submissions are pushed
• re-try push: all submissions are pushed, different versions visible on Central.
• pull Kasia's original All Widgets form with v1.17.4: none empty submissions folders visible in sd
• push Kasia's original All Widgets: all submissions are sent (2tries of push are needed when it has been pulled with 1.17.4)
• try to push form with an instance with xml media file: it still crashes briefcase @lognaturel are you sure that it should be fixed now?
• push form with multiple submissions version: the latest version is a published one

[lognaturel]

It’s looking like we’re almost there!

I clicked on the push but nothing happened

Is it possible the form on Central is a draft and not ending up getting published? What if you publish the draft? I will give this a try hopefully tomorrow.

[mmarciniak90]

Is it possible the form on Central is a draft and not ending up getting published?

@lognaturel I looked at this case once again and I realized that the attached data are complex so I tried the same case with a much more simple form and the result was correct.
It looks like an issue with strange media in form. There are xml files in form media. This is my special form 1648.zip

I pulled this form from Aggregate and from form directory. In both cases, nothing happened when I clicked on the 'Push' button. But still, the released version is able to push it and create a draft version.

[kkrawczyk123]

and more cases have been verified:

cmd push to Central form with submissions of different versions
validation on push encrypted form ui and cmd
cmd push to Central form pulled from Aggregate
cmd push to Central form pulled from Central
cmd push to Central form with submissions of different versions and media
cmd push form with media and submissions with media attached
cmd push form with submissions of different versions when there is already published form on Central
cmd push form with submissions of different versions when there is already draft form on Central
cmd pull from odk folder, cmd push to Central
cmd form pulled with 1.17.4 and pushed: only form known by briefcase and it's submissions are cmd pushed
cmd re-try push: all submissions are pushed, different versions visible on Central.
cmd pull Kasia's original form, and cmd push Kasia's original form
cmd push without specifying form id
I honestly don't have ideas for more, we brainstormed together with @mmarciniak90
@lognaturel I still can reproduce the issue with Briefcase freeze when a submission has xml attachment. Could you please take a look at that once again?

[lognaturel]

• Look at pushing 1648.zip which works on v1.17.4
• Freeze when a submission has xml attachment

The cases look quite complete. I'll look at those two remaining issues ASAP. Thank you!

[lognaturel]

I believe that both of you were running into the issue where form definitions used as media were causing conflicts. I had updated the method to look for form definitions but didn't update the call. It should now be fixed.

[kkrawczyk123]

We have confirmed that issues have been fixed. We have verified both via ui and cmd and also did quick regression check. We both agree that it's ready to go!
Verified on Ubuntu and MacOS.

@getodk-bot unlabel "needs testing"
@getodk-bot label "behavior verified"