apt-get source fails to fetch sourcedeb

[tfoote]

It's failing this job consistently: http://build.ros.org/job/Kbin_uW32__catkin__ubuntu_wily_i386__binary/3/console

# BEGIN SUBSECTION: get sourcedeb
00:00:34.425 Invoking '/usr/bin/python3 /tmp/ros_buildfarm/scripts/wrapper/apt-get.py source --download-only --only-source ros-kinetic-catkin=0.7.0-3wily'
00:02:30.086 Invoking 'apt-get source --download-only --only-source ros-kinetic-catkin=0.7.0-3wily'
00:02:30.086 Reading package lists...
00:02:30.086 Building dependency tree...
00:02:30.086 Reading state information...
00:02:30.086 Need to get 207 kB of source archives.
00:02:30.086 Get:1 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (dsc) [1,008 B]
00:02:30.086 Get:2 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (tar) [198 kB]
00:02:30.086 Get:3 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (diff) [8,620 B]
00:02:30.086 Fetched 207 kB in 0s (0 B/s)
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.dsc  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.debian.tar.xz  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch some archives.
00:02:30.086 
00:02:30.086 Failed to fetchInvocation failed due to the following known error conditions: , Hash Sum mismatch
00:02:30.086 
00:02:30.086 Reinvoke 'apt-get source' (2/10) after sleeping 9 seconds
00:02:30.086 Invoking 'apt-get source --download-only --only-source ros-kinetic-catkin=0.7.0-3wily'
00:02:30.086 Reading package lists...
00:02:30.086 Building dependency tree...
00:02:30.086 Reading state information...
00:02:30.086 Skipping already downloaded file 'ros-kinetic-catkin_0.7.0.orig.tar.gz'
00:02:30.086 Need to get 9,628 B of source archives.
00:02:30.086 Get:1 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (dsc) [1,008 B]
00:02:30.086 Get:2 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (diff) [8,620 B]
00:02:30.086 Fetched 9,624 B in 0s (0 B/s)
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.dsc  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.debian.tar.xz  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch some archives.
00:02:30.086 
00:02:30.086 Failed to fetchInvocation failed due to the following known error conditions: , Hash Sum mismatch
00:02:30.086 
00:02:30.086 Reinvoke 'apt-get source' (3/10) after sleeping 11 seconds
00:02:30.086 Invoking 'apt-get source --download-only --only-source ros-kinetic-catkin=0.7.0-3wily'
00:02:30.086 Reading package lists...
00:02:30.086 Building dependency tree...
00:02:30.086 Reading state information...
00:02:30.086 Skipping already downloaded file 'ros-kinetic-catkin_0.7.0.orig.tar.gz'
00:02:30.086 Need to get 9,628 B of source archives.
00:02:30.086 Get:1 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (dsc) [1,008 B]
00:02:30.086 Get:2 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (diff) [8,620 B]
00:02:30.086 Fetched 9,624 B in 0s (0 B/s)
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.dsc  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.debian.tar.xz  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch some archives.
00:02:30.086 
00:02:30.086 Failed to fetchInvocation failed due to the following known error conditions: , Hash Sum mismatch
00:02:30.086 
00:02:30.086 Reinvoke 'apt-get source' (4/10) after sleeping 13 seconds
00:02:30.086 Invoking 'apt-get source --download-only --only-source ros-kinetic-catkin=0.7.0-3wily'
00:02:30.086 Reading package lists...
00:02:30.086 Building dependency tree...
00:02:30.086 Reading state information...
00:02:30.086 Skipping already downloaded file 'ros-kinetic-catkin_0.7.0.orig.tar.gz'
00:02:30.086 Need to get 9,628 B of source archives.
00:02:30.086 Get:1 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (dsc) [1,008 B]
00:02:30.086 Get:2 http://repositories.ros.org/ubuntu/building/ wily/main ros-kinetic-catkin 0.7.0-3wily (diff) [8,620 B]
00:02:30.086 Fetched 9,624 B in 0s (0 B/s)
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.dsc  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch http://repositories.ros.org/ubuntu/building/pool/main/r/ros-kinetic-catkin/ros-kinetic-catkin_0.7.0-3wily.debian.tar.xz  Hash Sum mismatch
00:02:30.086 
00:02:30.086 E: Failed to fetch some archives.
00:02:30.086 
00:02:30.086 Failed to fetchInvocation failed due to the following known error conditions: , Hash Sum mismatch
00:02:30.086 
00:02:30.086 Reinvoke 'apt-get source' (5/10) after sleeping 15 seconds
00:02:30.086 Invoking 'apt-get source --download-only --only-source ros-kinetic-catkin=0.7.0-3wily'

[dirk-thomas]

This is not possible because the script runs as non-root.

[dirk-thomas]

I doubt the problem is due to a "bad" apt-get update and it is fixable by repeating it. It looks to me more like that the repeated build of the sourcedeb has put the apt repo in a bad state. Not sure how that happened and how to recover though.

[tfoote]

Hmm, yeah the issue with running w/o sudo is a problem. should we revert #249 or extend it to detect if we have sudo available? (Edit: I just saw #251 with the reversion)

I agree that we probably got the apt-repo into a bad state at some point. We should probably change the job to force an apt-get update during the last stage of the dockerfile build instead.

@dirk-thomas do you know why the above linked job lost it's history? The link above is now broken and if you go to the job itself it shows zero builds. (I just triggered it manually to see how it runs now. I think the apt-get update will be forced due to it being a new day)

[dirk-thomas]

I already reverted #249 in #250.

The job already enforce an apt-get update, see https://github.com/ros-infrastructure/ros_buildfarm/blob/master/ros_buildfarm/templates/release/binarydeb_create_task.Dockerfile.em#L50-L52

From a clean Docker I can apt-get source catkin without a problem but the build farm job (http://build.ros.org/view/Ksrc_uX/job/Kbin_uX64__catkin__ubuntu_xenial_amd64__binary/1/console#console-section-11) is unsuccessful. I haven't figured out why yet.

I deleted all Kinetic jobs repeatedly in order to verify that #251 actually works and the reconfigure job has established the job dependencies when it finishes.

[dirk-thomas]

Happened again on a debinc release:

• http://build.ros.org/job/Kbin_uX64__opencv3__ubuntu_xenial_amd64__binary/9/
• http://build.ros.org/job/Kbin_uX32__opencv3__ubuntu_xenial_i386__binary/9/

[dirk-thomas]

At the point where the build farm invoked apt-src it has "dropped" its root privileges and can't invoke apt-get update anymore. The problem is similar as #170 (comment) and as far as I can tell a problem in apt-get as it pretended to have finished successfully.

I would close this as "wontfix" since I don't see a way to address the problem.