Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nonexistant blocks clog up bitswap #830

Closed
whyrusleeping opened this issue Feb 26, 2015 · 10 comments
Closed

Nonexistant blocks clog up bitswap #830

whyrusleeping opened this issue Feb 26, 2015 · 10 comments
Labels
kind/bug A bug in existing code (including security flaws) topic/bandwidth reduction Topic bandwidth reduction topic/bitswap Topic bitswap

Comments

@whyrusleeping
Copy link
Member

If you request a block that does not exist in the network, bitswap will keep it in its wantlist and expend a significant amount of effort into finding that block. I'm not sure what the best way to solve this is, because what if the requested block is just really far away on a computer with a spotty internet connection? At what point is it acceptable to give up the search?

@whyrusleeping whyrusleeping added kind/bug A bug in existing code (including security flaws) topic/bitswap Topic bitswap labels Feb 26, 2015
@jbenet
Copy link
Member

jbenet commented Feb 27, 2015

great question. i think this should be done by callers-- i.e. if the gateway fails to find it, and the ctx expires, it should cancel it from bitswap. (not sure what a good way to cancel would be-- maybe binding it to a context?)

@whyrusleeping
Copy link
Member Author

alright, ill keep thinking about this.

it might be good to have an ipfs bitswap wantlist clear command to work with this stuff too.

@jbenet
Copy link
Member

jbenet commented Feb 27, 2015

ipfs bitswap wantlist clear sgtm

@whyrusleeping
Copy link
Member Author

This issue is also partly why our idle bandwidth is so high

@whyrusleeping
Copy link
Member Author

I want to bump this issue, mars had an idle bandwidth usage of 300KB/s, and had several keys in its wantlist that (i beleive) it couldnt find. Restarting the daemon lowered the bandwidth usage dramatically

@jbenet
Copy link
Member

jbenet commented Mar 9, 2015

@whyrusleeping it should be a matter of expiring the context, right? maybe the entries in the wantlist should have a list of contexts (or a refcount), and if all are cancelled, it's removed

@whyrusleeping
Copy link
Member Author

Yeah, we essentially just need an Unwant command on bitswap to be called upon context cancel in GetBlocks

@jbenet jbenet added the topic/bandwidth reduction Topic bandwidth reduction label Mar 28, 2015
@jbenet jbenet modified the milestone: Resource Constraints Mar 29, 2015
@whyrusleeping
Copy link
Member Author

couldnt dial mars this morning, ssh'ed in to check on it, seemed to be running fine, except running ipfs bitswap wantlist gives this:

whyrusleeping@mars ~> ipfs bitswap wantlist
QmV8ZdRddYMyoEtqE8LYDMUKWyp8xWNJ97vpPj7Es6GXQi
QmYNhsGhACvNFX16C25EQMB2kmh8dL1WM3G1tsrHzKcWSF
QmUF7QFNrEoNcWVUQqbvF2tQnQr1q5CWxLraUkrANcCfxE
Qma2WVnzgruHVDhbrRW5ah4JPL2xkoQ4ne9A5q6awzeAcN
QmdgaogTCo7nUoWUxhsJgKAwMYH5WTtbVB4VhKS3t5B4jc
QmcHdazu6sVrDXDpsMGkAWcrKbnjk7gVb5G1wZgf3JY4ei
QmXTJxsse2Tf9PFcM6KtTH7UQKUHnNM9UUm7hSnhNxdB3w
QmVUdwX5VUeSn78MfZNEp9KwEmnrZnVdMGV93pV5wdY5iW
QmTXKZVfm883YAsop5zWQqeLquFGRBy8KjLgVhcPGEFym5
QmZDgTTbLMEPRk8wr4NaiSqM69M366UCcBDNW6kXfGKSdu
QmQekNy6oPqrVUKPiFmwjMGsfVfo6by92ZrVvjp5JnKmHh
QmZbdePmvcviNQwUAgBK7kCrf3pwbWAMHr4E1cWBiM5yzd
QmS1VjqKvtej9uXbqjAHKGck7h3bnn3zgyQwhim5sQkKhY
QmZ6Je4hqEWaeGXdhu397eZVsfsVouWmHLE2F3S9LRZCHT
QmQ3k7gAe3r3ooD4yCp5YTPkjK22NsgYFC3kZqLnFkgu48
QmZATvSpp8MuTcZMhbGzQZMJVnf6mqrcH9bnBZHZ2xyEQ7
QmTwcCSvVk3JDEAmKDNf5v3rUEK9r6xAaNMF8mRH6kqf29
QmP6KBwWgyrxXkaLqdugzQve3LNCWZHfvZvLVnDZQ2YM8R
QmdkHzbP7G7fKYp4vMyA1WC2cePkRqHRgRWGA9KmiXFB9H
Qmf3fDrgSjC6yAEkSkFDvCaqymeGieom8JS9bdFUC7pQkx
QmPJwkrmXoDZY1bMXghUtqaUPgjaQzW7ixDiUuB8YAymiF
QmabtF3tdmEwE3jrAUbwq7y7dsVZKUTZtnVa9K73pzorBT
QmX19aczJRtiQdoZGYLaeRQZkd6PuMaFokWPJh8NGVTb9a
QmW5ngHkBnGpCnAETjswKfg4fdnMKKfu8Af9Ef7iMApYEf
QmZ9UhkbdhT1kRrxqEMX8iExa1Tq9CDL9YauZnhFydUz2N
QmNyFoQtMaNdf1CxKhG4EmyaPJX1MfSENpryHYuBVGqn1c
QmU3bmtrkkNarhM1PVwTMizvGv5BkS9EDGBeFvNc6hcJWR
QmV5NqLCsCLw7tjYPCTk5LhgNQRVkR2papc7A2YefweGRp
QmRd8pBgQYGn6yVQ2jsnimFxn3bf8U14BwhzyoUTHeDe85
QmZjs8eFMy7ox6m5LwRgWCkaAsZR6mpboY1ezkqabmqPvW
QmXPeD6A5a3qp5u9e7EUeedAkoMw3Dn3kzBFVEHLt5C1p3
Qmb3eXo3m5PDV1Kzxiok62qPR5HMYmZfi18oUyKDU3EtgB
Qmahs9B35mMFxTTrtDgAgRyj2b6wZzVmkGQYtEuKYDJwUD
QmaDhRAwSAhpirQNB6UXkuN9bozWcoFanuP7Y7D4qhWFcy
QmZb2ig6ojQqwUa4dr7mLVSZGrHxW8QKrRsc8fZSdqVKTC
QmbK4iHHcbTGobkX92mQ9mnyBKpHp5gL3BJiV4CDZRQpzN
QmcPcqHGD5gYThGD1VEEP5DQH4hWmGnUvFq6Nz5XvpU3y4
QmfVpYwrfTJdXokzegS8EA3qrX7VktQoRRMfvBAXtkMRWx
QmNnquZZwsMNp8mYrLibchYcpxcL6UibQxPZjZfL496jvP
QmajAqmjW4ggmwzJ3eSNVhLhPBLT3XDPWfVH9S9Nx91DMD
QmVbgpxo4NPNrpFQfCGuJpNQpCrJTFi7eo2vPho1aDmLEb
QmNW5ostom12n2w2dqkAn59ojxY76sP5u1GAoB2M17CpfH
Qma9ANbZBsYuDNbYPoDvMCrekr1DjMLVATLKV81rmSbxgt
QmTmif4hLERXqeAfuCqnw4Xo7SdKdRgLk2rvSvp7oPpQu8
QmQZBM7J3LA3pRbWdtV5Jx1CzHoR3GDF4xMb51YUiDSnDm
QmRAv7Khiya8D3KgTTz1kUdVfXLEnnAHbz6pkTwzpy8AeT
QmUdHHayLiR7X2nhtd6p59hxB8kUaH9XTJbLzGyZfers7m
QmUh5PErTEXgRQzZe5BKYRyqQaKHQbxK7d4ykrcyLqgEGE
QmQTorRviiWKpxdfymLyerUJEkL5xBC4GfbsqEzEc5kBR8
QmSK1qKo8e4YFDX1PL8g71ufh5pHuNSpLrAKMtxTp4S7Ke
QmXNJKTbeSq6xHc18WDru54WEGSwwYPP9T5VEcBZDmybxz
QmXuQKDDjHCjmw7TPhZjYeUCLywWmA1Rc4UP3913Huvn8J
QmSMobdGd78MpuaGxJCrT2DYyPmiBGi1HsaqphwJuPqCx5
QmRDrMbhsJ1dvqsENvZQ4ZBMPEpwi8hpckVKkYv9y2LFa8
QmaF97yQ2LwSk1riKrZEzNjsKRPSJgZwSS8hxjGqFP5qrS
QmXsV9pBTfyaHqgUbkFW4YaerRjhi8TignzVFD6eMMgn9C
QmQDdH8fcE6GtGq6ktRwEishBaaS5yjUo1iCQFfR93uNi7
QmavMG4MRBj6KzAuqmrRCDQt95PLPgzR16i1Hh4BKK9xMg
QmSSM8UKXz7yTFwuJdXVhqFzf16yZ92TYS7EFVi8v95WUk
QmW2KgbCQPC8ax8nhDdpAKoaZgUSu5Y4os9Qg6bttSZ1jk
QmRfYMAiFemg7HE5TmXSzNpc3VTYGJnQxBoUugr6YV3wzc
QmdxJD4hySRxziSEx11XJFE2Krec8WeDLWBPFAD2WbnqR4
QmXMRP41gYuLSwrnXZEvz4KqntJ66HNkZ6iguG9hUw5KW5
QmQjTrcsKt7uFgPM9oMGGYvPqgYCZB6DMkTyQdytSmwJwS
QmemqW1QQ9wifhjWCFCJPEf3FyoqH4rkgt82YfCPeNPkGP
QmWCM5XSb4RQSphyE7g7kXqfJZYM821ZmbAXzmFM1XwY1e
QmTBVgp5spKTYHuCithRRZouCR57vqfABWM6M4bMjhEjir
QmdcSWF6LQX9q3Qqa2EPX9swrLt9oqbPGh3KZCTZ59Qn27
QmaB5mkqFtXs9zkzgMRD4bme3Vn71WFwPzcAYfComJHvUk
QmW1edJgimiyXs7XcRtBeTSW4QQBwVCrBczDKuruwLb8Do
QmS3UhD5eNaa42t6VPFmxq2e1P3Nm8FPR7SmbEyv76M7fb
QmR9uVQ4XmSas1JV7AjsGBtKjDrH3m28EoFSkjics5ceF3
QmX71zSjtYNkR4E8aBRBuCejXfFyUm5UujnCTRfzEyPVXy
QmUGWf4dwKoTFGShYtqpEtQBDHNSQRA2grLouTfm7KRFgy
QmYTY1gzQwZ4fE3bEM7hhsuYgd55ADKXU54HhUBTvC4fzc
QmU7R7g4Fq6p4Mtx7cNUGmUnPk9YRAWQJNXrFMbKNLkbNZ
Qmarh2LDxULtMvDjMigacuQMqgL1tZr1icExg6XvXUFX99
QmRRn6sKBYoVX5o6YjQMAsYnS1TPgztBBj7m2K9Cco1Q73
QmXADBpgV3eETtJjn9umyP4pJDGsCLxEDquCtkKokFFduz
QmYmUGUo4MVahGbpMmdm65eXMFZKPx8waZNm4Fsd1Nq4Rj
QmYMK75mvxvUeBsnLgscY8F8vgHA2Gh4VsBGyko9ByWJ5R
QmZxrFDjJjZQTfC7KarvnBbUtqgZKmrYLKrxFEpUf5m4UA
QmWKSymsPdcmqmSCWDtQCsZJAQhy4tHM6976hBZCJf6WHb
QmVRf16XCBTTVjk85W7V8Li8SCrqQ53wxTozwiV7xoGY4f
Qmdmq2DTHGx8F63GP1ZUMQLJDaibW1ecArQPBSmBARWMWz
QmahoT6tbMwkSUzzCvkcmuyvkbM25nuaVowZfWKmQmMGoD
QmfWmdHsUXoDG9ExHVgMzyeFDRKgXAoirxqwJyg6z3BDu5
QmfShtGicknFYJMVsaJdUPWXSthhtQhfajag4E3LH64Bo9
QmXoyaNtK9w4tZC1yVdHWewumrYitooqKa3g29sayMXU5L
QmdPJbcLbntELeugEMCYG1eH8bps3FTo6pAZbgqTSmBzU7
QmapMPt2b2rmxQfKmr9stnvodfDwPN3LVfhMCCZp9S3my8
QmT4UcpjvKSP66qcWF2GVCMnhov6LkCws4oEHRSfAdF6EX
QmSa9F1YvYMSPaPGrXtwgeBf9BoyoeM32rZBfZNifD4bYK
QmP8JYCTPKUUsnjGSHbZoTL3C96WWGhZK1RTKnxRvUn3bi
QmU7hkgDUZJQwKMvFAX4k771QJrxLu6wZkezGCYtosUA2j
QmTdpQ1ndPUYWzRbWozHSfQa94NnEgVH2oPyuH4ejxJpBJ
QmeqJNHXV9SNfESfxvBf91PJmhQECrmMPM7PUwN4cwNoR9
QmXhw9B9rCS8pW3ak89d8oQPLmDwTy2oU9DhUMVxSbTc2j
QmYbm36KCNgqyfEa2FMfAF4GM53qtPkgwNftcf2GpQY6wn
QmbCqFt9MMJYeNkmuT75ftbDBitaLsmq5in2xWurTCsVyp
QmaiS3EEyoiixojEmaKnjUkAHY8QDb86npRWP2cCqJnE3Q
QmafZKV5UKXYWs6bvqB8kd26Xh8DibK6p6vCW98Qw3vaCh
QmPftbRn4kqbEHz9thgvSVprbN5RdWWYmqeUDEuarVP1DM
QmT7Q7qDaKtvzuwiA7TfiWZemX4khfTYMSEREp7ThWjy7Q
QmZY7BXC88xqt9co5oebNZfUjcRM5jTgvg5agA9GQ36Lfb
QmXXVr9tMUJ2qWcxE3ZUdvoVFuc7J7CMDhBvTsbWY3bXbV
QmZZpevoURZDDR5XnBGKYYbAam9SdjFNV4KEgLbHWX83x6
QmRGZ2yrwUrn6HNCt3qLi28hxcVePk6gCnZmkm9Qfo7LzD
QmczSnvYoep9jbFrhYkf7VMWFTt53JvFVT4B3nPbqE8x6f
QmbFt7wQJoJ4UaEJRC2h7z9Lgeme19AeZHuVuY5jBwnETZ
QmUrXBcnNa3uwVPf7QHgxwZvYdrRvFwLkfpCzQMmFitbh3
QmaCUmXydb8544zmvBFyQKE6AdyNRPgihE4u9NuuVWswBs
QmVbwChxcqhhmLaz4cdhxKgwVe5avcDfQQBFf3QRu52MeC
Qmf5Chi7tBi5VvXeQr6gfGQDwH6961mUXhwfp2SWnvwixW
QmQYdNT7G94TG2Qph4Dhf6rKeX7AKPbUn7P8kTFatqRjmc
QmUwnbjzTLHNdAEd5DVKRvjYUpK8H592Tv5dHvLhcWRVZe
QmcfcoThni8Wdf6VF8C3vN92vUQEbr7zy7eZqSHiruATyQ
QmT2cx2V98HrkJvPxREU4nf8ugXLN6rfRHod6qQ8oMCZzs
QmP2aKNRcJwLjueYgaJpvKZa4XgamWfAxWdbNHC6DPnX89
QmZT7Gxv5cgukj3s86FrXcRi6kmkWqmKuQj4Bbyq2x2e3a
QmPoTBEJobtDZHB4rf6ZR6Cq8TTf4PWKqmsT4vXx3yuheC
QmShLeAbpHgWfwXHLvZqW5zFjfZ9F4yMMgg2YM87PbFJyX
QmTE4v4aD9K7gN1Hqr3SZKzEUQ8PK88orDs4r85xgJqUSP
QmbucUttyjMDWMHShf9YmQ5nXADBQzA7gNvCav8tvzC5sF
QmS9d6ErWYq69EWKCavq4YLjNYisEKf86HCCVXfSVNvQFW
QmNrcbB6gH2iBKUXDByEpUT4KGGmHyfxqzNayjJcrdt5zd
QmNVkcG8WMShdWboQFsTMiS4CjZBPTtwtz1hq2FNDDyp3b
QmXxBBmAnviiTPCZQjE4XeWxLmzKEQ7nGyjyrYiPH4NGWX
Qmby8rc7WK6i4JuNemXyeNJGTtbdtR2ZLw3dfeDXfoiLXr
QmU8UdKufBVrjUPm8gUR8g9vtyrCt5d7hTpx3zPU5VkTw1
QmWaqSxhGsUBsnqoafhY2WxikzLMnKJ8gvoBwf6jvGM1Pk
QmWgbecxc9Tnn5uW6cXiDA1EEWUhJxvVFt8nNBQU7JSejE

I think this issue needs some higher priority

@jbenet
Copy link
Member

jbenet commented Apr 9, 2015

oof. agreed.

@dylanPowers dylanPowers added backlog and removed ready labels May 4, 2015
@daviddias daviddias removed the backlog label Jan 2, 2016
@whyrusleeping
Copy link
Member Author

this is fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) topic/bandwidth reduction Topic bandwidth reduction topic/bitswap Topic bitswap
Projects
None yet
Development

No branches or pull requests

4 participants