Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shapefiles corrupting - objects and attributes becoming switched #21356

Closed
qgib opened this issue Aug 31, 2015 · 22 comments
Closed

Shapefiles corrupting - objects and attributes becoming switched #21356

qgib opened this issue Aug 31, 2015 · 22 comments
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Crash/Data Corruption Digitizing Related to feature digitizing map tools or functionality High Priority

Comments

@qgib
Copy link
Contributor

qgib commented Aug 31, 2015

Author Name: Jamie Portman (Jamie Portman)
Original Redmine Issue: 13298
Affected QGIS version: 2.10.1
Redmine category:digitising


We've had recent experiences with one particular shapefile, where randomly some aspects of editing either object geometry or attributes is resulting in some line features dissapearing and the attributes of the one that is lost becoming attached to other objects in the file (and therefore their attributes lost).
On at least one occasion this behaviour has been related to using the split too (but we can't be sure it always is).
Have also seen some vertices of an object (that wasn't the one being edited) all of a sudden move to infinity (either south or east!) and no matter how far out we zoom, we never see the end of the line.

When attributes are getting mixed with other objects, it is often the record above or below in the dbf.

I have read some other forum issues detailing similar issues, but they often refer to the layer being edited by multiple people - this is definately NOT the case here. It was open by only one person.

We also never saw any "_packed" dbfs as others have reported.


@qgib
Copy link
Contributor Author

qgib commented Sep 9, 2015

Author Name: Saber Razmjooei (@saberraz)


Is the file located on a network drive?
Is your file topologically correct? I have seen this happened in the past when there is a row in dbf and no geometry for the data.


  • status_id was changed from Open to Feedback
  • priority_id was changed from High to Low

@qgib
Copy link
Contributor Author

qgib commented Sep 10, 2015

Author Name: Jamie Portman (Jamie Portman)


Yes the files were on a network drive.

Objects were topologically correct (no errors when running "check geometry validity").
The files were ONLY ever edited in QGIS, so if somehow the row in dbf & geometry were out of sync, then it could only have been QGIS that has somehow caused it...??

@qgib
Copy link
Contributor Author

qgib commented Sep 11, 2015

Author Name: Saber Razmjooei (@saberraz)


Could you try on the local drive?

@qgib
Copy link
Contributor Author

qgib commented Sep 16, 2015

Author Name: Jamie Portman (Jamie Portman)


It was a very random issue and we never got to the bottom of what was even causing it.
I did have another shapefile that when I used the split-line tool it went a little crazy (nodes moving to infinity) - but not so much the switching of attributes with geometry.
I can try it on local drive and see if the same thing happens...

@qgib
Copy link
Contributor Author

qgib commented Sep 17, 2015

Author Name: Jamie Portman (Jamie Portman)


I found the shapefile and object that was having ONE of the issues (not the attributes and objects getting switched) but doing insane corruption changes when cut with the 'Split Object' tool.

See screen grabs and shapefile example attached.
When the line is cut using the Split Object tool in v2.10 - the result is as per the screen shots - with vertices moving to infinity and the object becomes un-selectable.
Exactly the same process in v2.81 results in the line being split exactly as expected.

The line feature probably originated from a CAD DXF, however has been edited, cut, moved and changed in QGIS many times over a period of months. It was only recently when I cut the line using v2.10 that this issue occurred. Clearly whatever is causing the issue was most likely present all along, but some change in v2.10 is now causing very unexpected results.

Interesting, if I copy and paste the feature and move it, then sometimes the copied feature behaves normally and other times not...!

Certainly some similar behaviour issues as seen in the other file that had issues, but I have yet to be able to replicate the switching objects and attributes problem.

*Note. in the screenshots, the original feature is the one on the left. The right one is a copied feature.


  • 9080 was configured as Test_DodgyRoad.zip
  • 9076 was configured as BeforeSplit.JPG
  • 9077 was configured as Split.JPG
  • 9078 was configured as AfterSplit.JPG
  • 9079 was configured as AfterSplit_ZoomOut.JPG

@qgib
Copy link
Contributor Author

qgib commented Sep 17, 2015

Author Name: Jamie Portman (Jamie Portman)


Opps - forgot to mention, the above tests were done with the file on the local drive.

@qgib
Copy link
Contributor Author

qgib commented Oct 12, 2015

Author Name: Saber Razmjooei (@saberraz)


  • status_id was changed from Feedback to Open

@qgib
Copy link
Contributor Author

qgib commented Oct 28, 2015

Author Name: Harrissou Santanna (@DelazJ)


I wonder why this issue is set to low priority while it causes data corruption.

@qgib
Copy link
Contributor Author

qgib commented Oct 28, 2015

Author Name: Jean-Pierre Vinadia (Jean-Pierre Vinadia)


I have quite the same problem with QGIS 2.12.
I work on a local drive.
This problem does not appears with the 2.8.3 version.
When I modify lignes (cut, copy/paste for exemple) and then (or before) rename the field (un champ de la table attributaire) the names of some objets move in an other object.

Some times, I can just realise that this the problem has apeared only when I register the modifications. So, I also think that it is a high priority because it can completly "destroy" the file.

(I apology for my english. I hope that you can understand me)


  • 9230 was configured as Destructuration_données_table_attributaire.zip
  • 9229 was configured as Screen_Shot.JPG

@qgib
Copy link
Contributor Author

qgib commented Nov 7, 2015

Author Name: Giovanni Manghi (@gioman)


This seems really similar to #21768

can someone provide a clear sequence of steps to allow us replicate the issue in a consistent way?


  • status_id was changed from Open to Feedback
  • priority_id was changed from Low to High

@qgib
Copy link
Contributor Author

qgib commented Nov 24, 2015

Author Name: Jean-Pierre Vinadia (Jean-Pierre Vinadia)


The sequence (cf the attached file)
I open "Boucle fichier origine"
Save as "boucle fichier modifié"
and then cut lines.
At this moment, The attributes move from one objetc to another. (see screenshot in my messaqge above)

This problem appears again this morning, working with another computer in an other office with QGIS 2.12 .
Unfortunately, i cannot repeat this disfonction on my own comuter yet.
It seams to appear in a random way.

The sequence this morinig. All done with Tablemanager
Open a file with 10 fields.
Delete 9 fields keeping only the field "NOM". Rename it in NAME and add fields (ID, LONGUEUR)
The name of the object appears not on the right object.


  • 9347 was configured as fichier_destructuré.zip

@qgib
Copy link
Contributor Author

qgib commented Dec 4, 2015

Author Name: Jean-Pierre Vinadia (Jean-Pierre Vinadia)


The problem appaers again with an other file. With QGIS 2.12.0

It's really a very very big problem. Fortunately, I had a save of my work. But this makes me unconfortable while using QGIS. I'm yet afraid of the results of my manipulations.

Il just delete one column caled "TYPE_FORM" and the codes of my table has been totaly destructured.
So th original file is 'Balisage"
The corrupted file is Balisage BUG 04-12


  • 9374 was configured as BUG_QGIS.zip

@qgib
Copy link
Contributor Author

qgib commented Dec 4, 2015

Author Name: Harrissou Santanna (@DelazJ)


Hi,
How do you delete the field: with "Table Manager" plugin or with the Core tool available in attribute table?

@qgib
Copy link
Contributor Author

qgib commented Dec 4, 2015

Author Name: Jean-Pierre Vinadia (Jean-Pierre Vinadia)


With Tablemanager.

@qgib
Copy link
Contributor Author

qgib commented Dec 27, 2015

Author Name: Giovanni Manghi (@gioman)


Jean-Pierre Vinadia wrote:

The problem appaers again with an other file. With QGIS 2.12.0

It's really a very very big problem. Fortunately, I had a save of my work. But this makes me unconfortable while using QGIS. I'm yet afraid of the results of my manipulations.

Il just delete one column caled "TYPE_FORM" and the codes of my table has been totaly destructured.
So th original file is 'Balisage"
The corrupted file is Balisage BUG 04-12

I can't see any incongruence between the two table of attributes. I also tested by removing the "TYPE_FORM" field from 'Balisage' and got no corruption of data whatsoever. Anyway if you still get problems by removing fields with table manager (but not QGIS itself) then report the issue in the table manager bug tracker, as is a 3rd party plugin.

@qgib
Copy link
Contributor Author

qgib commented Dec 27, 2015

Author Name: Giovanni Manghi (@gioman)


Jean-Pierre Vinadia wrote:

I have quite the same problem with QGIS 2.12.
I work on a local drive.
This problem does not appears with the 2.8.3 version.
When I modify lignes (cut, copy/paste for exemple) and then (or before) rename the field (un champ de la table attributaire) the names of some objets move in an other object.

Just tested again on master and can't replicate.

I think we still miss here a clear way (detailed steps) to replicate the issue.

@qgib
Copy link
Contributor Author

qgib commented Dec 27, 2015

Author Name: Giovanni Manghi (@gioman)


Jamie Portman wrote:

I found the shapefile and object that was having ONE of the issues (not the attributes and objects getting switched) but doing insane corruption changes when cut with the 'Split Object' tool.

See screen grabs and shapefile example attached.
When the line is cut using the Split Object tool in v2.10 - the result is as per the screen shots - with vertices moving to infinity and the object becomes un-selectable.
Exactly the same process in v2.81 results in the line being split exactly as expected.

just tested on master and can't replicate this behavior.

Detailed steps or a screencast would help.

@qgib
Copy link
Contributor Author

qgib commented Dec 27, 2015

Author Name: Giovanni Manghi (@gioman)


I also can't replicate the original issue in the description, the one about the mixing/switching attributes.

Please test again on QGIS master and report back.

@qgib
Copy link
Contributor Author

qgib commented Jan 10, 2016

Author Name: Jamie Portman (Jamie Portman)


I have tested the Split Object Tool issue (with lines going crazy when cut) in the latest nightly dev version and 2.12 - both versions appear to have fixed this issue. It didn't occur with every shapefile, but I found one that it was happening with and it still happens in 2.10 but not from 2.12 onwards :)

What is still happening in the latest nightly dev build (2.13) is deleted objects are not removed properly when saved. I deleted half the objects from a Shapefile, saved edits and closed. Re-open in QGIS and they do not appear, however, open the file in ArcGIS and they are all still there. If a 'save-as' of the Shapefile is done in 2.13, then they are properly removed when the file is opened in ArcGIS.

The mixing/switching of attributes was a VERY random problem with one particular file. We have not been able to replicate or reproduce the exact sequence of steps with any other file. I suspect it has something to do with these 'ghost' deleted features, as when you split a line it effectively deletes that object and creates two new ones. It only even seemed to happen after having done a split of a feature, but it often wasn't discovered until hours later when much more editing had been done, so we couldn't be sure.

@qgib
Copy link
Contributor Author

qgib commented Jan 13, 2016

Author Name: Giovanni Manghi (@gioman)


What is still happening in the latest nightly dev build (2.13) is deleted objects are not removed properly when saved. I deleted half the objects from a Shapefile, saved edits and closed. Re-open in QGIS and they do not appear, however, open the file in ArcGIS and they are all still there. If a 'save-as' of the Shapefile is done in 2.13, then they are properly removed when the file is opened in ArcGIS.

this unfortunately is a known issue, that should be fixes in 2.14

#19349

The mixing/switching of attributes was a VERY random problem with one particular file. We have not been able to replicate or reproduce the exact sequence of steps with any other file. I suspect it has something to do with these 'ghost' deleted features, as when you split a line it effectively deletes that object and creates two new ones. It only even seemed to happen after having done a split of a feature, but it often wasn't discovered until hours later when much more editing had been done, so we couldn't be sure.

this probably is a consequence of the above.

Can we close this issue then, as fixed in 2.12/master?

@qgib
Copy link
Contributor Author

qgib commented Jan 13, 2016

Author Name: Jamie Portman (Jamie Portman)


I think so.
It can always be reopened if we ever see it again and can figure out a repeatable cause.

Thanks.

@qgib
Copy link
Contributor Author

qgib commented Feb 25, 2016

Author Name: Jürgen Fischer (@jef-n)


  • status_id was changed from Feedback to Closed
  • resolution was changed from to fixed/implemented

@qgib qgib added Bug Either a bug report, or a bug fix. Let's hope for the latter! High Priority Digitizing Related to feature digitizing map tools or functionality Crash/Data Corruption labels May 25, 2019
@qgib qgib added this to the Future Release - High Priority milestone May 25, 2019
@qgib qgib closed this as completed May 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Crash/Data Corruption Digitizing Related to feature digitizing map tools or functionality High Priority
Projects
None yet
Development

No branches or pull requests

1 participant