Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.944530294s FAIL (2025-07-26 23:36:10 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.94455432s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [.X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X....] duration: 51.44731203s FAIL (2025-07-26 23:36:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.447357839s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [X.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.229200411s FAIL (2025-07-26 23:35:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 83793813s) [seg_pos={0 0}, upload=57550]: Begin upload segment... Jul 26 23:34:58.761315385 (59h59m8.483878505s) [noise=true , address=79.127.220.97:4361 , piece_id=DC4GICBG22IHKMD3GXG4IO7L7FKCBIH7XLUACA2AIM34DZJCK36Q, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.761545877 (59h59m8.484108917s) [seg_pos={0 0}, upload=57550]: Done begin upload segment. Jul 26 23:34:58.761562498 (59h59m8.484125528s) [noise=true , address=79.127.205.245:4228 , piece_id=PF6VLPNIEUDE6Y7QNC6QZZGY776M5RSKAJPEY7B4IIPYWSIWHPBQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.762269681 (59h59m8.484832721s) [noise=true , address=79.127.205.233:4623 , piece_id=XBT3CBXMIDAPSTNVXN5OHEIALM3VUI5Y3OFIQ3U24WP5QWFCBTAA, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.762641353 (59h59m8.485204393s) [noise=true , address=45.91.116.5:4306 , piece_id=POZ4B22NDHINVYRLKJ62DTES5IMTU6UYIRP6ZSQRV2PDXDIXUM4A, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.762882834 (59h59m8.485445874s) [noise=true , address=104.36.82.56:4260 , piece_id=GJGOKLIKBZYQD44H5EPVFGUHVVPRDVZU2ZFDBPGAWZ7EMY32ZERA, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.763105630 (59h59m8.48566867s) [noise=true , address=77.90.31.13:4643 , piece_id=Z4O3LKZ4HNGORL6ZUNEI5TDMN32UWCH6X2BBVJDQL2OK377FCFJQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.763313378 (59h59m8.485876418s) [noise=true , address=77.90.31.10:4538 , piece_id=LMAZ53M334E7HKRPEDRNFIH3OHYXXAHATA6TA53HU4E53VIKMIVQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.763522859 (59h59m8.4860859s) [noise=true , address=77.90.31.19:4875 , piece_id=ZK4HH2D3HBP7QG7P3W6PX6FROJ6HZEXDSHLYCSVAVZGL3N266UYQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.763727070 (59h59m8.48629011s) [noise=true , address=104.36.82.46:4492 , piece_id=C52DEKYXEKXSRSQMIGV7USTHUEAF22L3BHBLIWSSI43JTCBQOJSA, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.763939617 (59h59m8.486502657s) [noise=true , address=79.127.205.234:4610 , piece_id=NHX6RK34GW4TJYTX4QMXX3ZPUGFV5FZMT7REBISZ57YHUYCV7YSQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.764131955 (59h59m8.486694995s) [noise=true , address=156.146.43.225:4449 , piece_id=K32ZB2PQWYNQCKBLO7XRS25TPSGNZMEL4RWJO7LJH2R4RL4POFHA, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.764477626 (59h59m8.487040666s) [noise=true , address=104.36.82.47:4521 , piece_id=F3SX3P52WHKMRGVEWYYV4N322OIC3CT2L3WIL7HOZC6EADYKHHAQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.764779804 (59h59m8.487342844s) [noise=true , address=79.127.219.51:4212 , piece_id=JNE6O7TBG57C7OQYS667RIZUMTHBQRKNNHJLLTSJXEPMUIXTLBEQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.765046043 (59h59m8.487609083s) [noise=true , address=104.36.82.55:4225 , piece_id=NOMMMDUX4HYTRPHPMRDAU6P2XMJAZR5RLJPMFFTA7KSOLBNCFCCQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.765296793 (59h59m8.487859833s) [noise=true , address=104.36.82.53:4131 , piece_id=OXLHVRU5K3OH3MKQIRVRPU3NYIY6XPKJM463PY4PGDXSZWJBAJKA, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.765553063 (59h59m8.488116103s) [noise=true , address=79.127.219.38:4018 , piece_id=HTQDXV5IVAMPLGXMDPDFFEUOSPGNYQKZLMVN3SEXOGGNIDXFLD4A, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.765843599 (59h59m8.488406639s) [noise=true , address=79.127.219.47:4048 , piece_id=4RNV5Y6SJH6O4OPPR3AJPUJCUFMDAHKHCJZHQH2TQCAUITAA6JYA, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:58.766103586 (59h59m8.488666626s) [noise=true , address=45.91.116.14:4206 , piece_id=ZYMW3D6B3RHDXLXEOSBPUXALJKYTC24OCVGHXKO4TYFDOCRLMRUQ, seg_pos={0 0}, upload=57550]: Uploading piece... Jul 26 23:34:59.818456586 (59h59m9.541019626s) [noise=true , address=77.90.31.10:4538 , piece_id=LMAZ53M334E7HKRPEDRNFIH3OHYXXAHATA6TA53HU4E53VIKMIVQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: storage node overloaded, request limit: 2000 Jul 26 23:34:59.824295398 (59h59m9.546858438s) [noise=true , address=79.127.220.97:4361 , piece_id=DC4GICBG22IHKMD3GXG4IO7L7FKCBIH7XLUACA2AIM34DZJCK36Q, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 23:34:59.873688692 (59h59m9.596251732s) [upload=57550]: Next returned nil segment Jul 26 23:34:59.873704983 (59h59m9.596268023s) [upload=57550]: Waiting for error group managing segments... Jul 26 23:35:00.176049259 (59h59m9.898612299s) [noise=true , address=79.127.205.245:4228 , piece_id=PF6VLPNIEUDE6Y7QNC6QZZGY776M5RSKAJPEY7B4IIPYWSIWHPBQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:00.179266314 (59h59m9.901829354s) [noise=true , address=79.127.205.234:4610 , piece_id=NHX6RK34GW4TJYTX4QMXX3ZPUGFV5FZMT7REBISZ57YHUYCV7YSQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:00.192767976 (59h59m9.915331006s) [noise=true , address=79.127.205.233:4623 , piece_id=XBT3CBXMIDAPSTNVXN5OHEIALM3VUI5Y3OFIQ3U24WP5QWFCBTAA, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:00.414009358 (59h59m10.136572398s) [noise=true , address=45.91.116.5:4306 , piece_id=POZ4B22NDHINVYRLKJ62DTES5IMTU6UYIRP6ZSQRV2PDXDIXUM4A, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:1QuFvRW9wGKdKRXPK8UdR8uqTm53CnHaF9CsUz2bUh2Ptw9r1F, address:45.91.116.5:4306): protocol: storage node overloaded, request limit: 2000 Jul 26 23:35:00.737490475 (59h59m10.460053515s) [noise=true , address=79.127.219.38:4018 , piece_id=HTQDXV5IVAMPLGXMDPDFFEUOSPGNYQKZLMVN3SEXOGGNIDXFLD4A, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:01.837872919 (59h59m11.560435969s) [noise=true , address=79.127.219.51:4212 , piece_id=JNE6O7TBG57C7OQYS667RIZUMTHBQRKNNHJLLTSJXEPMUIXTLBEQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:04.763150549 (59h59m14.485713589s) [noise=true , address=45.91.116.14:4206 , piece_id=ZYMW3D6B3RHDXLXEOSBPUXALJKYTC24OCVGHXKO4TYFDOCRLMRUQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:05.621317352 (59h59m15.343880392s) [noise=true , address=79.127.219.47:4048 , piece_id=4RNV5Y6SJH6O4OPPR3AJPUJCUFMDAHKHCJZHQH2TQCAUITAA6JYA, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:07.840346324 (59h59m17.562909354s) [noise=true , address=156.146.43.225:4449 , piece_id=K32ZB2PQWYNQCKBLO7XRS25TPSGNZMEL4RWJO7LJH2R4RL4POFHA, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:08.899334237 (59h59m18.621897267s) [noise=true , address=77.90.31.13:4643 , piece_id=Z4O3LKZ4HNGORL6ZUNEI5TDMN32UWCH6X2BBVJDQL2OK377FCFJQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:09.090316294 (59h59m18.812879335s) [noise=true , address=77.90.31.19:4875 , piece_id=ZK4HH2D3HBP7QG7P3W6PX6FROJ6HZEXDSHLYCSVAVZGL3N266UYQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: <nil> Jul 26 23:35:57.628247986 (60h0m7.350811016s) [noise=true , address=104.36.82.46:4492 , piece_id=C52DEKYXEKXSRSQMIGV7USTHUEAF22L3BHBLIWSSI43JTCBQOJSA, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: write tcp 188.166.225.170:39784->104.36.82.46:4492: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:57.628339891 (60h0m7.350902971s) [noise=true , address=104.36.82.55:4225 , piece_id=NOMMMDUX4HYTRPHPMRDAU6P2XMJAZR5RLJPMFFTA7KSOLBNCFCCQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: write tcp 188.166.225.170:38734->104.36.82.55:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:57.628483997 (60h0m7.351047038s) [noise=true , address=104.36.82.53:4131 , piece_id=OXLHVRU5K3OH3MKQIRVRPU3NYIY6XPKJM463PY4PGDXSZWJBAJKA, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:1uhyc5ozrF9ndeEnQXtjc9o3tsYiAnW1dzeuHV3kv1JS4T4a8C, address:104.36.82.53:4131): protocol: write tcp 188.166.225.170:42942->104.36.82.53:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:57.628568519 (60h0m7.351131569s) [noise=true , address=104.36.82.56:4260 , piece_id=GJGOKLIKBZYQD44H5EPVFGUHVVPRDVZU2ZFDBPGAWZ7EMY32ZERA, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:35:57.629238320 (60h0m7.35180135s) [noise=true , address=104.36.82.47:4521 , piece_id=F3SX3P52WHKMRGVEWYYV4N322OIC3CT2L3WIL7HOZC6EADYKHHAQ, seg_pos={0 0}, upload=57550]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:104.36.82.47:4521): protocol: write tcp 188.166.225.170:59658->104.36.82.47:4521: use of closed network connection; context deadline exceeded Jul 26 23:35:57.629258800 (60h0m7.35182184s) [seg_pos={0 0}, upload=57550]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:35:57.855397681 (60h0m7.577960721s) [upload=57550]: Done uploading segments... Jul 26 23:35:57.855413361 (60h0m7.577976391s) [upload=57550]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.22921084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [.X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.240529188s FAIL (2025-07-26 23:35:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== meout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482170676 (59h59m33.204733706s) [seg_pos={0 0}, upload=57549]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:35:23.742315641 (59h59m33.464878671s) [upload=57549]: Done uploading segments... Jul 26 23:35:23.742347543 (59h59m33.464910583s) [upload=57549]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Jul 26 23:35:24.621181507 (59h59m34.343744548s) [upload=57557]: Starting upload Jul 26 23:35:24.621286257 (59h59m34.343849297s) [upload=57557]: Done starting upload Jul 26 23:35:24.621401387 (59h59m34.343964428s) [upload=57557]: Uploading segments... Jul 26 23:35:24.621561554 (59h59m34.344124594s) [upload=57557]: Got next segment. Inline: false Jul 26 23:35:24.621601802 (59h59m34.344164842s) [upload=57557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:35:24.621607021 (59h59m34.344170061s) [upload=57557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:35:24.876132365 (59h59m34.598695405s) [seg_pos={0 0}, upload=57557]: Begin upload segment... Jul 26 23:35:24.876218590 (59h59m34.59878163s) [seg_pos={0 0}, upload=57557]: Done begin upload segment. Jul 26 23:35:24.876308453 (59h59m34.598871493s) [noise=true , address=45.91.116.7:4009 , piece_id=LC25OV2ZEGR5TVYUQOGF7U4KU4O76JD3EGWAGDW4NS54IMEZUDCQ, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.876341245 (59h59m34.598904285s) [noise=true , address=45.91.116.13:4153 , piece_id=IL2C2ASYLNQKDSE3OKZEG7BVWP4BG7WZZ54DLOVGTCHKIWWYJHAQ, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.876819269 (59h59m34.599382299s) [noise=true , address=104.36.82.14:4221 , piece_id=UN6WT25XSWRNGJVD4ZPAMGDCQNF2XV2KU3BPWEVJZ5RRFERH24UQ, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.876838306 (59h59m34.599401346s) [noise=true , address=104.36.82.5:4839 , piece_id=B43YAKKZJWMUIQYRO7WRVFF5QIDRVEZM7KJERNBW4UXOUFAOTILQ, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.877166354 (59h59m34.599729394s) [noise=true , address=45.91.116.4:4937 , piece_id=62TLFM7LK6KSFQNUUW5C6RQM7UI4ZOBJX5JSFW3XC6Y73NU72B4A, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.877181111 (59h59m34.599744152s) [noise=true , address=45.91.116.12:4225 , piece_id=PRZYLROGKHG4T3KZI77GDIN5ZROUVCKQWGIRGX5JDEOXWS2HQBEA, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.877449175 (59h59m34.600012215s) [noise=true , address=104.36.82.12:4149 , piece_id=W2S4Y6LL6A2ROTISODWRNTVR3NLXO27VWZAK7LQGQ4AJUPGSW37Q, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.877476256 (59h59m34.600039296s) [noise=true , address=45.91.116.6:4973 , piece_id=RDXUS7Z6MYNIH472R7532SCI4WTEOEXKRVPX6PFZOKJZIVQTUZ2A, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.877897602 (59h59m34.600460642s) [noise=true , address=77.90.31.13:4628 , piece_id=JYKZXBEO7ETYFIOWCZEWCJZY26MD7MZWTNQG5VETW3RE32P6DNHA, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.878086674 (59h59m34.600649734s) [noise=true , address=104.36.82.3:4767 , piece_id=Z7V45ZVKQSHLICV5XJSLN3CZFDIRS5FOLGUTBMUXAQSUMXW6WJ2Q, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.878484716 (59h59m34.601047766s) [noise=true , address=77.90.31.12:4590 , piece_id=GQMUOBUCXNHJJWZIE647UYNWKCZRREN3SQXKGAY2CHJM5OPX7ATA, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.878805239 (59h59m34.601368279s) [noise=true , address=104.36.82.8:4971 , piece_id=AKJYCVRJU6O346OUSLU55BLUJXPHLH7OQMOCTDUEOOMEAZZJXMYA, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.879016183 (59h59m34.601579223s) [noise=true , address=45.91.116.11:4118 , piece_id=GARZVV4QGXIWQ3VD7FGOT24ZM4JRFIQAU34B3L5NSQGTESSGBAAQ, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.879214131 (59h59m34.601777171s) [noise=true , address=104.36.82.4:4801 , piece_id=756BC2NBKHDMMADIUD7INXFYPVUKXYN4IBSDYSJLJB3LFZTHQGFA, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.879522020 (59h59m34.60208506s) [noise=true , address=45.91.116.12:4227 , piece_id=V5DHWDMYT22EA27AVZ3MQVG5YHT7U4A7C2QZST4A3M4ECCKMYW3A, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:24.879766759 (59h59m34.602329799s) [noise=true , address=45.91.116.5:4299 , piece_id=BJSJHMVKDGBFVWPILGNZZZN276V5GITHELXXTX27CYPEHVMAG4OQ, seg_pos={0 0}, upload=57557]: Uploading piece... Jul 26 23:35:26.619456370 (59h59m36.342019411s) [upload=57557]: Next returned nil segment Jul 26 23:35:26.619474696 (59h59m36.342037736s) [upload=57557]: Waiting for error group managing segments... Jul 26 23:35:26.906728358 (59h59m36.629291398s) [noise=true , address=45.91.116.13:4153 , piece_id=IL2C2ASYLNQKDSE3OKZEG7BVWP4BG7WZZ54DLOVGTCHKIWWYJHAQ, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:26.974686854 (59h59m36.697249894s) [noise=true , address=45.91.116.7:4009 , piece_id=LC25OV2ZEGR5TVYUQOGF7U4KU4O76JD3EGWAGDW4NS54IMEZUDCQ, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:26.975749177 (59h59m36.698312217s) [noise=true , address=77.90.31.12:4590 , piece_id=GQMUOBUCXNHJJWZIE647UYNWKCZRREN3SQXKGAY2CHJM5OPX7ATA, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:26.976514911 (59h59m36.699077951s) [noise=true , address=45.91.116.6:4973 , piece_id=RDXUS7Z6MYNIH472R7532SCI4WTEOEXKRVPX6PFZOKJZIVQTUZ2A, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:27.022299602 (59h59m36.744862642s) [noise=true , address=45.91.116.5:4299 , piece_id=BJSJHMVKDGBFVWPILGNZZZN276V5GITHELXXTX27CYPEHVMAG4OQ, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:27.023269928 (59h59m36.745832958s) [noise=true , address=45.91.116.11:4118 , piece_id=GARZVV4QGXIWQ3VD7FGOT24ZM4JRFIQAU34B3L5NSQGTESSGBAAQ, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:27.034280485 (59h59m36.756843515s) [noise=true , address=45.91.116.12:4227 , piece_id=V5DHWDMYT22EA27AVZ3MQVG5YHT7U4A7C2QZST4A3M4ECCKMYW3A, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:27.040605096 (59h59m36.763168136s) [noise=true , address=45.91.116.4:4937 , piece_id=62TLFM7LK6KSFQNUUW5C6RQM7UI4ZOBJX5JSFW3XC6Y73NU72B4A, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:27.083795992 (59h59m36.806359032s) [noise=true , address=45.91.116.12:4225 , piece_id=PRZYLROGKHG4T3KZI77GDIN5ZROUVCKQWGIRGX5JDEOXWS2HQBEA, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:28.585420975 (59h59m38.307984015s) [noise=true , address=77.90.31.13:4628 , piece_id=JYKZXBEO7ETYFIOWCZEWCJZY26MD7MZWTNQG5VETW3RE32P6DNHA, seg_pos={0 0}, upload=57557]: Done uploading piece. err: <nil> Jul 26 23:35:53.971788355 (60h0m3.694351395s) [noise=true , address=104.36.82.8:4971 , piece_id=AKJYCVRJU6O346OUSLU55BLUJXPHLH7OQMOCTDUEOOMEAZZJXMYA, seg_pos={0 0}, upload=57557]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:53.971844623 (60h0m3.694407663s) [noise=true , address=104.36.82.14:4221 , piece_id=UN6WT25XSWRNGJVD4ZPAMGDCQNF2XV2KU3BPWEVJZ5RRFERH24UQ, seg_pos={0 0}, upload=57557]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:58528->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:53.971882855 (60h0m3.694445905s) [noise=true , address=104.36.82.5:4839 , piece_id=B43YAKKZJWMUIQYRO7WRVFF5QIDRVEZM7KJERNBW4UXOUFAOTILQ, seg_pos={0 0}, upload=57557]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:53.971906451 (60h0m3.694469491s) [noise=true , address=104.36.82.4:4801 , piece_id=756BC2NBKHDMMADIUD7INXFYPVUKXYN4IBSDYSJLJB3LFZTHQGFA, seg_pos={0 0}, upload=57557]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:53.971939975 (60h0m3.694503015s) [noise=true , address=104.36.82.3:4767 , piece_id=Z7V45ZVKQSHLICV5XJSLN3CZFDIRS5FOLGUTBMUXAQSUMXW6WJ2Q, seg_pos={0 0}, upload=57557]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:53.971957889 (60h0m3.694520929s) [noise=true , address=104.36.82.12:4149 , piece_id=W2S4Y6LL6A2ROTISODWRNTVR3NLXO27VWZAK7LQGQ4AJUPGSW37Q, seg_pos={0 0}, upload=57557]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:50460->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:53.971975834 (60h0m3.694538864s) [seg_pos={0 0}, upload=57557]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:35:54.210456169 (60h0m3.933019209s) [upload=57557]: Done uploading segments... Jul 26 23:35:54.210480105 (60h0m3.933043145s) [upload=57557]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.240583673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235510015s FAIL (2025-07-26 23:35:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.235531177s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [..X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.262358108s FAIL (2025-07-26 23:35:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 4:54.119593530 (59h59m3.84215657s) [upload=57549]: Starting upload Jul 26 23:34:54.119649728 (59h59m3.842212768s) [upload=57549]: Done starting upload Jul 26 23:34:54.119712978 (59h59m3.842276018s) [upload=57549]: Uploading segments... Jul 26 23:34:54.119818621 (59h59m3.842381651s) [upload=57549]: Got next segment. Inline: false Jul 26 23:34:54.119843268 (59h59m3.842406308s) [upload=57549]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:34:54.119847496 (59h59m3.842410536s) [upload=57549]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:34:54.370121004 (59h59m4.092684044s) [seg_pos={0 0}, upload=57549]: Begin upload segment... Jul 26 23:34:54.370162833 (59h59m4.092725864s) [seg_pos={0 0}, upload=57549]: Done begin upload segment. Jul 26 23:34:54.370184405 (59h59m4.092747435s) [noise=true , address=104.36.82.7:4909 , piece_id=LWFLLKD3JLS5R24CRD3SZI6M6HEIGQKPRV57PNUOXEPNYAOX36SA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.370239911 (59h59m4.092802951s) [noise=true , address=45.91.116.10:4082 , piece_id=QEH5W3KZRFCIC7MVJPZXWNY5WZJTTLLCGB4KHA4LHUXHQQMDOJVA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.370570804 (59h59m4.093133844s) [noise=true , address=45.91.116.12:4226 , piece_id=7M7NIB6VS4U4VXZ4TRCODR4USJ6TF5JT3GTOYFOLX6256EERI5WA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.370722254 (59h59m4.093285295s) [noise=true , address=45.91.116.8:4047 , piece_id=AXL5NOVIUW36NZTJSVWWIHEALGXQBTJSSSM336CNMGE6WJKNK7MA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.370866210 (59h59m4.09342924s) [noise=true , address=104.36.82.14:4223 , piece_id=Y3XGRMELKUH75LQT5O6B4MA7QSMXKJ3Y7QDEFJPUHZNRXOKTCK5A, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.370987922 (59h59m4.093550952s) [noise=true , address=104.36.82.13:4185 , piece_id=2WFECOV4PDTJVUJODGUCWZK3EWADFLFB3EVNQELWP3KMKGGLDDAQ, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371237620 (59h59m4.09380066s) [noise=true , address=45.91.116.7:4009 , piece_id=Q7TQVCSRXOOGNCWIXEAVOXYC7XSE6EHNRNTVPL5JCG7W4BZZGQ6Q, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371245105 (59h59m4.093808145s) [noise=true , address=45.91.116.9:4261 , piece_id=GRRJJUYBQVTP4QDAXC2TL4WAKFWAAAAKASSAAKXINVPW3CES2ITQ, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371453673 (59h59m4.094016703s) [noise=true , address=104.36.82.2:4730 , piece_id=V3Z3KS6CH4RGGVSE2MUAQTWICL4Q5JJVJFWY4UTMGZJE7FCOGVLA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371471398 (59h59m4.094034438s) [noise=true , address=104.36.82.6:4875 , piece_id=XM7QQKBMLP3SQI5YVHWJPHZRHV5CJJONWNS2XVFOZSTKE5MITRKQ, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371651973 (59h59m4.094215014s) [noise=true , address=45.91.116.12:4225 , piece_id=BZFZHZBPTBZTGPJ6NB32SQ6DLL7A5VPVJQ6IJ5HUITFF4A45RWKQ, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371710866 (59h59m4.094273906s) [noise=true , address=77.90.31.14:4663 , piece_id=73BNJM3QNNCOVQ7WUWW45YUCFI5Q7VMGSHEMDVGZZC3G7CGCU73Q, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371904837 (59h59m4.094467877s) [noise=true , address=104.36.82.8:4971 , piece_id=OWN4XNJYIUXCVOAISTQQFI4RK2M3ECDEJCMJ4XN4PBQ4IL7QJHOQ, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.371924766 (59h59m4.094487806s) [noise=true , address=104.36.82.4:4803 , piece_id=D2AXM4MXAC5IWZKQHY7GNG74DFDPF7OR2XMSYY3MNUE5DO4NBEQA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.372158332 (59h59m4.094721372s) [noise=true , address=104.36.82.9:4007 , piece_id=OUMRSJ5QXAXJAL2VWOFSEDYBZFP5RQWUXDXWU6AVC6PAN7VRCPQA, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:54.372252332 (59h59m4.094815372s) [noise=true , address=45.91.116.6:4975 , piece_id=VC5YWKLEQBKXPYUQT6SBKSZWQDSWLNLKDRKUUIOZY5DC4CCDHAVQ, seg_pos={0 0}, upload=57549]: Uploading piece... Jul 26 23:34:55.999862101 (59h59m5.722425141s) [upload=57549]: Next returned nil segment Jul 26 23:34:55.999880756 (59h59m5.722443796s) [upload=57549]: Waiting for error group managing segments... Jul 26 23:34:56.391010816 (59h59m6.113573856s) [noise=true , address=77.90.31.14:4663 , piece_id=73BNJM3QNNCOVQ7WUWW45YUCFI5Q7VMGSHEMDVGZZC3G7CGCU73Q, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.397393719 (59h59m6.119956759s) [noise=true , address=45.91.116.10:4082 , piece_id=QEH5W3KZRFCIC7MVJPZXWNY5WZJTTLLCGB4KHA4LHUXHQQMDOJVA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.404500046 (59h59m6.127063076s) [noise=true , address=45.91.116.12:4225 , piece_id=BZFZHZBPTBZTGPJ6NB32SQ6DLL7A5VPVJQ6IJ5HUITFF4A45RWKQ, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.464763298 (59h59m6.187326338s) [noise=true , address=45.91.116.12:4226 , piece_id=7M7NIB6VS4U4VXZ4TRCODR4USJ6TF5JT3GTOYFOLX6256EERI5WA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.495221504 (59h59m6.217784544s) [noise=true , address=45.91.116.7:4009 , piece_id=Q7TQVCSRXOOGNCWIXEAVOXYC7XSE6EHNRNTVPL5JCG7W4BZZGQ6Q, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.532021172 (59h59m6.254584212s) [noise=true , address=45.91.116.9:4261 , piece_id=GRRJJUYBQVTP4QDAXC2TL4WAKFWAAAAKASSAAKXINVPW3CES2ITQ, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.535146522 (59h59m6.257709592s) [noise=true , address=45.91.116.6:4975 , piece_id=VC5YWKLEQBKXPYUQT6SBKSZWQDSWLNLKDRKUUIOZY5DC4CCDHAVQ, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:34:56.535757892 (59h59m6.258320932s) [noise=true , address=45.91.116.8:4047 , piece_id=AXL5NOVIUW36NZTJSVWWIHEALGXQBTJSSSM336CNMGE6WJKNK7MA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: <nil> Jul 26 23:35:23.481964340 (59h59m33.20452738s) [noise=true , address=104.36.82.8:4971 , piece_id=OWN4XNJYIUXCVOAISTQQFI4RK2M3ECDEJCMJ4XN4PBQ4IL7QJHOQ, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.481991883 (59h59m33.204554923s) [noise=true , address=104.36.82.13:4185 , piece_id=2WFECOV4PDTJVUJODGUCWZK3EWADFLFB3EVNQELWP3KMKGGLDDAQ, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: write tcp 188.166.225.170:38040->104.36.82.13:4185: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482032541 (59h59m33.204595571s) [noise=true , address=104.36.82.14:4223 , piece_id=Y3XGRMELKUH75LQT5O6B4MA7QSMXKJ3Y7QDEFJPUHZNRXOKTCK5A, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482044644 (59h59m33.204607684s) [noise=true , address=104.36.82.6:4875 , piece_id=XM7QQKBMLP3SQI5YVHWJPHZRHV5CJJONWNS2XVFOZSTKE5MITRKQ, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:41486->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482081796 (59h59m33.204644836s) [noise=true , address=104.36.82.4:4803 , piece_id=D2AXM4MXAC5IWZKQHY7GNG74DFDPF7OR2XMSYY3MNUE5DO4NBEQA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482085703 (59h59m33.204648743s) [noise=true , address=104.36.82.2:4730 , piece_id=V3Z3KS6CH4RGGVSE2MUAQTWICL4Q5JJVJFWY4UTMGZJE7FCOGVLA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482120119 (59h59m33.204683159s) [noise=true , address=104.36.82.9:4007 , piece_id=OUMRSJ5QXAXJAL2VWOFSEDYBZFP5RQWUXDXWU6AVC6PAN7VRCPQA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:36226->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482156198 (59h59m33.204719238s) [noise=true , address=104.36.82.7:4909 , piece_id=LWFLLKD3JLS5R24CRD3SZI6M6HEIGQKPRV57PNUOXEPNYAOX36SA, seg_pos={0 0}, upload=57549]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:23.482170676 (59h59m33.204733706s) [seg_pos={0 0}, upload=57549]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:35:23.742315641 (59h59m33.464878671s) [upload=57549]: Done uploading segments... Jul 26 23:35:23.742347543 (59h59m33.464910583s) [upload=57549]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.262370801s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........................XXX.X..X.....XXXXX...XXX.X.....X.X..] duration: 13.252104497s FAIL (2025-07-26 23:35:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.252129125s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.934084868s FAIL (2025-07-26 23:35:11 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.934259212s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.42207882s FAIL (2025-07-26 23:35:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.422093338s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 54 out of 60 is failed (threshold=3) [XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236666332s FAIL (2025-07-26 23:34:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== , address=104.36.82.47:4551 , piece_id=CQLU5MBSK6DK2OV3S6TTXR6NQ4PAXEV2WO3N7OQCKMY3GDXEMHCA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.274234885 (59h58m7.996797925s) [noise=true , address=79.127.205.245:4239 , piece_id=N7L2MO7TCCWBTHW66HGUFNYFHUOT7MYT4AZKH3IIHYIMN4734SGQ, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.274596296 (59h58m7.997159336s) [noise=true , address=77.90.31.21:4882 , piece_id=GRWZAAH6474GOR2XS537R734ZVBS6BMOO2KEGEZGWJPRP5XBCZHQ, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.274857326 (59h58m7.997420366s) [noise=true , address=79.127.219.51:4212 , piece_id=ZIHF646NQUV63JRQFE6TF2OSKH7X4LVP4RRAYX4FHXN33BZVOS3A, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.275105811 (59h58m7.997668851s) [noise=true , address=104.36.82.54:4172 , piece_id=6NJOIJ6Y4RLAHYZFZB242KBT7TKERQDNAX57DOSFJKUSEMKGSFJA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.275349908 (59h58m7.997912949s) [noise=true , address=79.127.201.215:4354 , piece_id=6ORM6J2DI4GQV4GFMHIU2GGUUG4PBC6X2POGERQ7M2CGKB5IOFMQ, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.275558387 (59h58m7.998121427s) [noise=true , address=104.36.82.55:4201 , piece_id=RIOFDHDM5ZZ26ZAATJRBAXZNH7HGMHD5UORHSWFW3IGZQX6ZTD2Q, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.275803776 (59h58m7.998366816s) [noise=true , address=79.127.220.101:4041 , piece_id=SMVO27QGC52CJ7655ARMTVN6E76ZZOUNKGKKLC5JKLFMJHUJWHWQ, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.276018077 (59h58m7.998581117s) [noise=true , address=104.36.82.52:4097 , piece_id=OM6Z73NDR7FF67C4RC6W3GXO7LUAR7BRF4ZAXPTI7ZITS5LVELUQ, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.276252525 (59h58m7.998815565s) [noise=true , address=77.90.31.15:4727 , piece_id=62C2DKTOWWV56KQEGDYLRXDHGAZYSCBXX7AOGZ2FQGLQDXIY4E4A, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.276459952 (59h58m7.999022992s) [noise=true , address=79.127.201.214:4321 , piece_id=JYJONOMGW2XAJ5OOTGLNN5YRIK7OXKLZLCIANC53GLDRBOFOZJUA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.276717084 (59h58m7.999280124s) [noise=true , address=104.36.82.53:4121 , piece_id=FICV3L7RQZBVV7XMVNO5PD533OKZU6SBPTHPZJMRBPSC5SMYZHOA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.276982321 (59h58m7.999545361s) [noise=true , address=104.36.82.11:4145 , piece_id=Z53INGIGYP3W57APTXK4DQKZOISZSANGIIL4YNQX42XKVCYZDESA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.277179178 (59h58m7.999742218s) [noise=true , address=104.36.82.56:4240 , piece_id=M52TD72M24WMQUMYKV3WFRJ5CUV2QFY76DTA4Q3JQXF4TZEB4IWQ, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.277385022 (59h58m7.999948062s) [noise=true , address=79.127.220.98:4422 , piece_id=C6ROGYZTQ4HKSU4OTNEMWKTKQE737W52WVC52L7YYGS57O63UDIA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:58.277572941 (59h58m8.000135981s) [noise=true , address=104.36.82.9:4027 , piece_id=HPOXTLVQOOBK5UKM6WYP4CBUA3EQM4NCQ3LLIH5SNESD2XWVKTHA, seg_pos={0 0}, upload=57533]: Uploading piece... Jul 26 23:33:59.402988960 (59h58m9.125552001s) [upload=57533]: Next returned nil segment Jul 26 23:33:59.403017455 (59h58m9.125580495s) [upload=57533]: Waiting for error group managing segments... Jul 26 23:33:59.464147375 (59h58m9.186710415s) [noise=true , address=79.127.201.214:4321 , piece_id=JYJONOMGW2XAJ5OOTGLNN5YRIK7OXKLZLCIANC53GLDRBOFOZJUA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:127EgxjVRhAGcz1DJjE539rWnmDjsVTRFQc1esEZUyQ1yxGiCD5, address:79.127.201.214:4321): protocol: storage node overloaded, request limit: 2000 Jul 26 23:33:59.659816299 (59h58m9.382379359s) [noise=true , address=64.49.13.140:4508 , piece_id=HJP3EHDWMZETRUVOMYSWYWQTSFDXATSSAYTAY2GZ4Q4RVPWW5JLA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:00.314566984 (59h58m10.037130024s) [noise=true , address=77.90.31.21:4882 , piece_id=GRWZAAH6474GOR2XS537R734ZVBS6BMOO2KEGEZGWJPRP5XBCZHQ, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:00.353949163 (59h58m10.076512204s) [noise=true , address=77.90.31.15:4727 , piece_id=62C2DKTOWWV56KQEGDYLRXDHGAZYSCBXX7AOGZ2FQGLQDXIY4E4A, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:00.933669985 (59h58m10.656233025s) [noise=true , address=79.127.219.51:4212 , piece_id=ZIHF646NQUV63JRQFE6TF2OSKH7X4LVP4RRAYX4FHXN33BZVOS3A, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:02.621544229 (59h58m12.344107279s) [noise=true , address=79.127.220.101:4041 , piece_id=SMVO27QGC52CJ7655ARMTVN6E76ZZOUNKGKKLC5JKLFMJHUJWHWQ, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:03.794743110 (59h58m13.51730615s) [noise=true , address=79.127.220.98:4422 , piece_id=C6ROGYZTQ4HKSU4OTNEMWKTKQE737W52WVC52L7YYGS57O63UDIA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:06.318255241 (59h58m16.040818271s) [noise=true , address=79.127.205.245:4239 , piece_id=N7L2MO7TCCWBTHW66HGUFNYFHUOT7MYT4AZKH3IIHYIMN4734SGQ, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:43.100921730 (59h58m52.82348476s) [noise=true , address=79.127.201.215:4354 , piece_id=6ORM6J2DI4GQV4GFMHIU2GGUUG4PBC6X2POGERQ7M2CGKB5IOFMQ, seg_pos={0 0}, upload=57533]: Done uploading piece. err: <nil> Jul 26 23:34:57.159588899 (59h59m6.882151929s) [noise=true , address=104.36.82.54:4172 , piece_id=6NJOIJ6Y4RLAHYZFZB242KBT7TKERQDNAX57DOSFJKUSEMKGSFJA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: write tcp 188.166.225.170:45234->104.36.82.54:4172: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159645367 (59h59m6.882208397s) [noise=true , address=104.36.82.53:4121 , piece_id=FICV3L7RQZBVV7XMVNO5PD533OKZU6SBPTHPZJMRBPSC5SMYZHOA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: rpc: dial tcp 104.36.82.53:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159690363 (59h59m6.882253393s) [noise=true , address=104.36.82.55:4201 , piece_id=RIOFDHDM5ZZ26ZAATJRBAXZNH7HGMHD5UORHSWFW3IGZQX6ZTD2Q, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: write tcp 188.166.225.170:59514->104.36.82.55:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159699610 (59h59m6.88226265s) [noise=true , address=104.36.82.56:4240 , piece_id=M52TD72M24WMQUMYKV3WFRJ5CUV2QFY76DTA4Q3JQXF4TZEB4IWQ, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:129QqkFEzZdtzQ6HAKxPWxDNgDVUNJJLH59oRVXnHExrwkDECyf, address:104.36.82.56:4240): protocol: rpc: dial tcp 104.36.82.56:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159726883 (59h59m6.882289923s) [noise=true , address=104.36.82.46:4505 , piece_id=CU2O7UODA7MZ6ZYSIDEALEJUN5NMYKYPFUDU6YYXNCPKBL5XVWVA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: rpc: dial tcp 104.36.82.46:4505: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159762971 (59h59m6.882326011s) [noise=true , address=104.36.82.11:4145 , piece_id=Z53INGIGYP3W57APTXK4DQKZOISZSANGIIL4YNQX42XKVCYZDESA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: rpc: dial tcp 104.36.82.11:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159768432 (59h59m6.882331472s) [noise=true , address=104.36.82.47:4551 , piece_id=CQLU5MBSK6DK2OV3S6TTXR6NQ4PAXEV2WO3N7OQCKMY3GDXEMHCA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:1UhJvSMtNqJDak4TM7eUgz1N7yCn4ChmRRTe7nXdffTNsS1pch, address:104.36.82.47:4551): protocol: rpc: dial tcp 104.36.82.47:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159807637 (59h59m6.882370677s) [noise=true , address=104.36.82.9:4027 , piece_id=HPOXTLVQOOBK5UKM6WYP4CBUA3EQM4NCQ3LLIH5SNESD2XWVKTHA, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:1SWk13XvkrCYwKcdZ46SK2THQvhsD6pf4hWJKgARztB9qwfMZU, address:104.36.82.9:4027): protocol: rpc: dial tcp 104.36.82.9:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159828618 (59h59m6.882391658s) [noise=true , address=104.36.82.52:4097 , piece_id=OM6Z73NDR7FF67C4RC6W3GXO7LUAR7BRF4ZAXPTI7ZITS5LVELUQ, seg_pos={0 0}, upload=57533]: Done uploading piece. err: ecclient: upload failed (node:1sMhTktqf2H4aZSyFV3m9Z6eMVxL2U52bvUHVHUh2gUCVp32AM, address:104.36.82.52:4097): protocol: write tcp 188.166.225.170:44978->104.36.82.52:4097: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:57.159856280 (59h59m6.882419311s) [seg_pos={0 0}, upload=57533]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:34:57.394272981 (59h59m7.116836021s) [upload=57533]: Done uploading segments... Jul 26 23:34:57.394300443 (59h59m7.116863483s) [upload=57533]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.236681831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [...X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232329276s FAIL (2025-07-26 23:34:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== ) [upload=57539]: Done starting upload Jul 26 23:34:23.694354607 (59h58m33.416917647s) [upload=57539]: Uploading segments... Jul 26 23:34:23.694559079 (59h58m33.417122119s) [upload=57539]: Got next segment. Inline: false Jul 26 23:34:23.694584879 (59h58m33.417147919s) [upload=57539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:34:23.694589167 (59h58m33.417152207s) [upload=57539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:34:23.970233738 (59h58m33.692796768s) [seg_pos={0 0}, upload=57539]: Begin upload segment... Jul 26 23:34:23.970275899 (59h58m33.692838939s) [seg_pos={0 0}, upload=57539]: Done begin upload segment. Jul 26 23:34:23.970294754 (59h58m33.692857794s) [noise=true , address=45.91.116.10:4083 , piece_id=VGHD4ZWA7VXEATEKFHUEFVGXCMQRR4SCDLQMLXRIQB4QAKC5DHMQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.970303421 (59h58m33.692866461s) [noise=true , address=104.36.82.2:4731 , piece_id=CK6SHNJ2QQVD7NIIEBOWN5JF5Q3XUL6HK6VEJVQMNDNZD76AEVYQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.970377943 (59h58m33.692940973s) [noise=true , address=45.91.116.14:4189 , piece_id=HRIK4BT5MIOKE5CIGHIXJI4565BI3L7CGKPX2FH3WFCLJ7D5XIHA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.970659413 (59h58m33.693222453s) [noise=true , address=104.36.82.14:4221 , piece_id=BQUV34MX6JGGFKILJPJY67ZIP36SHYZE5TE2M5MO2JJCXEDCSAEQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.970749284 (59h58m33.693312314s) [noise=true , address=104.36.82.12:4149 , piece_id=KCI7PEHU7MNKSKBVEU4XGZLKSX33Q55J3P7R6Z3G67IYECCRQNJA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.970991788 (59h58m33.693554828s) [noise=true , address=77.90.31.11:4554 , piece_id=X4BKV3MYB5VO5GRGREKBJRLCDV4HFM4PRE57MVOKLPKNCQB52MDA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.971056432 (59h58m33.693619472s) [noise=true , address=104.36.82.5:4838 , piece_id=QPC4GIUIJ5KV64IUHJWHUBEO3TVOWV7Q7YIF5OQQEI2RJLEUAZNQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.971246125 (59h58m33.693809165s) [noise=true , address=77.90.31.12:4590 , piece_id=VQZPYQINZ76PGRJUTIBEGXMI2BQ6SIJZ66SET2EPPP42LKCOZKQQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.971481666 (59h58m33.694044696s) [noise=true , address=77.90.31.18:4805 , piece_id=KDYDB66IDVHOCWW2UFDHL5HYDTIWE6GLKZH4NF24JG3HFQBYMQRA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.971644397 (59h58m33.694207487s) [noise=true , address=77.90.31.12:4592 , piece_id=IMY2ZNXBIYU2WU5PVRNSR6WOPWJ6UE2HIPQYGJW63757F5GKXCIA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.971711535 (59h58m33.694274565s) [noise=true , address=45.91.116.6:4973 , piece_id=NDHTIFEJB77TXPKVRNQ64UEYJYPWEJUI67RFNKZDJZPYFQ76QZGQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.972140597 (59h58m33.694703637s) [noise=true , address=104.36.82.11:4113 , piece_id=I3WB5JKXC5H7NTAOIQ63DMIYMFQUL2VXBINS5KDFTGBVQ4DEOGAQ, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.972439367 (59h58m33.695002407s) [noise=true , address=104.36.82.8:4970 , piece_id=SQMHHKN5WWBZKYPIAVXMJUMLW6BW4VNHT5YAAD4VIAZYS3VVUVCA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.972649460 (59h58m33.6952125s) [noise=true , address=45.91.116.14:4191 , piece_id=H255PWMUYDPTCYW2GZIJLPOYZU4NS2WCPGFCC5HJTRXQSFK6D5IA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.972851907 (59h58m33.695414947s) [noise=true , address=45.91.116.8:4048 , piece_id=2QRHRN4KKUFB5XIH377F5ZKGWXZVS32VNIMKCWZPX47V3CL4P6CA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:23.973058342 (59h58m33.695621382s) [noise=true , address=45.91.116.13:4153 , piece_id=HAUFFBSBPSK7UMTQ26BYU2J5OIVY5D3U53DDL6OWC7PJDL35SPDA, seg_pos={0 0}, upload=57539]: Uploading piece... Jul 26 23:34:25.713267760 (59h58m35.4358308s) [upload=57539]: Next returned nil segment Jul 26 23:34:25.713289751 (59h58m35.435852801s) [upload=57539]: Waiting for error group managing segments... Jul 26 23:34:26.005986940 (59h58m35.72854997s) [noise=true , address=45.91.116.10:4083 , piece_id=VGHD4ZWA7VXEATEKFHUEFVGXCMQRR4SCDLQMLXRIQB4QAKC5DHMQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:26.007546333 (59h58m35.730109373s) [noise=true , address=45.91.116.6:4973 , piece_id=NDHTIFEJB77TXPKVRNQ64UEYJYPWEJUI67RFNKZDJZPYFQ76QZGQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:26.069484250 (59h58m35.79204729s) [noise=true , address=45.91.116.8:4048 , piece_id=2QRHRN4KKUFB5XIH377F5ZKGWXZVS32VNIMKCWZPX47V3CL4P6CA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:26.074935479 (59h58m35.797498519s) [noise=true , address=45.91.116.14:4191 , piece_id=H255PWMUYDPTCYW2GZIJLPOYZU4NS2WCPGFCC5HJTRXQSFK6D5IA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:26.141003668 (59h58m35.863566708s) [noise=true , address=45.91.116.14:4189 , piece_id=HRIK4BT5MIOKE5CIGHIXJI4565BI3L7CGKPX2FH3WFCLJ7D5XIHA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:26.184455634 (59h58m35.907018664s) [noise=true , address=45.91.116.13:4153 , piece_id=HAUFFBSBPSK7UMTQ26BYU2J5OIVY5D3U53DDL6OWC7PJDL35SPDA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:36.055025389 (59h58m45.77758843s) [noise=true , address=77.90.31.12:4590 , piece_id=VQZPYQINZ76PGRJUTIBEGXMI2BQ6SIJZ66SET2EPPP42LKCOZKQQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:36.326102972 (59h58m46.048666013s) [noise=true , address=77.90.31.12:4592 , piece_id=IMY2ZNXBIYU2WU5PVRNSR6WOPWJ6UE2HIPQYGJW63757F5GKXCIA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: <nil> Jul 26 23:34:53.034348128 (59h59m2.756911168s) [noise=true , address=104.36.82.11:4113 , piece_id=I3WB5JKXC5H7NTAOIQ63DMIYMFQUL2VXBINS5KDFTGBVQ4DEOGAQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:53.034393154 (59h59m2.756956194s) [noise=true , address=104.36.82.8:4970 , piece_id=SQMHHKN5WWBZKYPIAVXMJUMLW6BW4VNHT5YAAD4VIAZYS3VVUVCA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:53.034470341 (59h59m2.757033381s) [noise=true , address=104.36.82.5:4838 , piece_id=QPC4GIUIJ5KV64IUHJWHUBEO3TVOWV7Q7YIF5OQQEI2RJLEUAZNQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:53.034538561 (59h59m2.757101602s) [noise=true , address=77.90.31.18:4805 , piece_id=KDYDB66IDVHOCWW2UFDHL5HYDTIWE6GLKZH4NF24JG3HFQBYMQRA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:32900->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:32900->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32900->77.90.31.18:4805: use of closed network connection Jul 26 23:34:53.034563860 (59h59m2.7571269s) [noise=true , address=104.36.82.12:4149 , piece_id=KCI7PEHU7MNKSKBVEU4XGZLKSX33Q55J3P7R6Z3G67IYECCRQNJA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 188.166.225.170:54188->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:53.034574311 (59h59m2.757137351s) [noise=true , address=104.36.82.14:4221 , piece_id=BQUV34MX6JGGFKILJPJY67ZIP36SHYZE5TE2M5MO2JJCXEDCSAEQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:53.034605009 (59h59m2.757168049s) [noise=true , address=104.36.82.2:4731 , piece_id=CK6SHNJ2QQVD7NIIEBOWN5JF5Q3XUL6HK6VEJVQMNDNZD76AEVYQ, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:53.034650306 (59h59m2.757213346s) [noise=true , address=77.90.31.11:4554 , piece_id=X4BKV3MYB5VO5GRGREKBJRLCDV4HFM4PRE57MVOKLPKNCQB52MDA, seg_pos={0 0}, upload=57539]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 188.166.225.170:51284->77.90.31.11:4554: use of closed network connection; context deadline exceeded Jul 26 23:34:53.034673570 (59h59m2.75723661s) [seg_pos={0 0}, upload=57539]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 23:34:53.265413045 (59h59m2.987976085s) [upload=57539]: Done uploading segments... Jul 26 23:34:53.265438454 (59h59m2.988001494s) [upload=57539]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.232352812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.969977171s FAIL (2025-07-26 23:34:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.970002149s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [..X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X...] duration: 27.850074474s FAIL (2025-07-26 23:34:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.8500982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.232337978s FAIL (2025-07-26 23:34:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.232358256s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........................XXX.X..X.....XXXXX...XXX.X.....X.X.] duration: 7.83064773s FAIL (2025-07-26 23:34:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.830664892s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........................XXX.X..X.....XXXXX...XXX.X.....X.X] duration: 30.000519856s FAIL (2025-07-26 23:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000539783s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.226566464s FAIL (2025-07-26 23:33:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 525550122s) [noise=true , address=104.36.82.10:4095 , piece_id=SNRYXZAJDTGPUFVQXLRLYBS64USKELHLHBHGOPFC2MXHTUUXWH6Q, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.803292937 (59h57m7.525855977s) [noise=true , address=104.36.82.55:4227 , piece_id=QNY42STGBS2QSRHT6IOE7OAI7CJFSDLIC2C2HCN3M4KSANETOA6A, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.803548345 (59h57m7.526111385s) [noise=true , address=77.90.31.16:4756 , piece_id=QMIBFB33ZC6V7V25GWC7TZ6ZHXRMBWBJGMGSAESS274ZIFSLKQ7Q, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.803906090 (59h57m7.52646912s) [noise=true , address=79.127.219.51:4212 , piece_id=5FVNLGVEXATFTWK3F3JU4TC5RRTYVJEDYYBCSW7KHS7GJIV3244A, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.804164755 (59h57m7.526727785s) [noise=true , address=79.127.201.215:4352 , piece_id=UQOX4NCOHWGBE4GW7DF4EJYZD5X6M55A6S47MS5EXLNUH2SDE2FQ, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.804454068 (59h57m7.527017108s) [noise=true , address=45.91.116.14:4206 , piece_id=IKJXOHH6MWLBWZPYLK5D7MGKIP7HEQLXIFL2LGOT37WSMV6WTUCA, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.804664571 (59h57m7.527227611s) [noise=true , address=104.36.82.47:4538 , piece_id=UM23T636SQMTYUDHJJG7GW256RHFIOPQJII2UZB5X5DMLUTAADAQ, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.804937414 (59h57m7.527500454s) [noise=true , address=104.36.82.11:4131 , piece_id=SMM5OA5FTW55P5IBDO7K3LZFSBWIL3RSVEWOMOOH7EOR5OJI5KHA, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.805185107 (59h57m7.527748147s) [noise=true , address=79.127.205.233:4633 , piece_id=HWW44LPFN74YZ7YHHCRB66Q7UCBUR7ZD4Z47B7O7SL7QZMVY74WA, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.805403034 (59h57m7.527966075s) [noise=true , address=104.36.82.52:4087 , piece_id=NOKKGLYW73K3XPTSHVOBS34STKKC7454RBW64ILQH6JY5G2ZZR4A, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.805589281 (59h57m7.528152321s) [noise=true , address=77.90.31.10:4535 , piece_id=3MV5OCU2KEB5XFROI6B3LJKYTPLRVXDFKKFV5PKCTY44EFUFVL5Q, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.805788472 (59h57m7.528351512s) [noise=true , address=104.36.82.53:4156 , piece_id=IQTE2HTTO6MQQD4DBTTVMN62H3STCQVTOJZZJXYRTA4XB46JRZFQ, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.806038921 (59h57m7.528601961s) [noise=true , address=79.127.205.245:4245 , piece_id=FZPOATQIYPQR525OXI2E2REZ3GQC6MWCOP6XG2WM72TL2SUD6J2A, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.806244374 (59h57m7.528807414s) [noise=true , address=79.127.205.246:4288 , piece_id=CILCJREJ5SS3CVWRJOM6O5CN4ZV5LMDREVSOV2D4M3XJDHXHLC3A, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.806426382 (59h57m7.528989422s) [noise=true , address=156.146.43.225:4434 , piece_id=2H6COFERB2APIIEPMI52KII7H4W3LFAVCEAWHSFAIR2PF3P3G7DQ, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.806607760 (59h57m7.52917079s) [noise=true , address=79.127.219.38:4017 , piece_id=32RKG5KRI5LYVTU6KXFSIJYJBG2F3D75RFXCHJSYOAMCOFLMPSWQ, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:57.806793063 (59h57m7.529356104s) [noise=true , address=104.36.82.46:4508 , piece_id=PWZAGWXHIXMWCMGK2B5AVWHLVAZ54DSTAOOA2JDRI5OMI2WXHMDA, seg_pos={0 0}, upload=57519]: Uploading piece... Jul 26 23:32:58.916560824 (59h57m8.639123864s) [upload=57519]: Next returned nil segment Jul 26 23:32:58.916577797 (59h57m8.639140837s) [upload=57519]: Waiting for error group managing segments... Jul 26 23:32:59.296974199 (59h57m9.019537229s) [noise=true , address=79.127.205.246:4288 , piece_id=CILCJREJ5SS3CVWRJOM6O5CN4ZV5LMDREVSOV2D4M3XJDHXHLC3A, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:32:59.310639194 (59h57m9.033202264s) [noise=true , address=79.127.205.233:4633 , piece_id=HWW44LPFN74YZ7YHHCRB66Q7UCBUR7ZD4Z47B7O7SL7QZMVY74WA, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:32:59.342678095 (59h57m9.065241135s) [noise=true , address=79.127.205.245:4245 , piece_id=FZPOATQIYPQR525OXI2E2REZ3GQC6MWCOP6XG2WM72TL2SUD6J2A, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:32:59.697094290 (59h57m9.41965733s) [noise=true , address=79.127.219.38:4017 , piece_id=32RKG5KRI5LYVTU6KXFSIJYJBG2F3D75RFXCHJSYOAMCOFLMPSWQ, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:32:59.866008154 (59h57m9.588571184s) [noise=true , address=77.90.31.10:4535 , piece_id=3MV5OCU2KEB5XFROI6B3LJKYTPLRVXDFKKFV5PKCTY44EFUFVL5Q, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:32:59.923011057 (59h57m9.645574097s) [noise=true , address=77.90.31.16:4756 , piece_id=QMIBFB33ZC6V7V25GWC7TZ6ZHXRMBWBJGMGSAESS274ZIFSLKQ7Q, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:33:00.361500149 (59h57m10.084063229s) [noise=true , address=79.127.219.51:4212 , piece_id=5FVNLGVEXATFTWK3F3JU4TC5RRTYVJEDYYBCSW7KHS7GJIV3244A, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:33:02.667260525 (59h57m12.389823565s) [noise=true , address=156.146.43.225:4434 , piece_id=2H6COFERB2APIIEPMI52KII7H4W3LFAVCEAWHSFAIR2PF3P3G7DQ, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:33:04.411305810 (59h57m14.13386885s) [noise=true , address=45.91.116.14:4206 , piece_id=IKJXOHH6MWLBWZPYLK5D7MGKIP7HEQLXIFL2LGOT37WSMV6WTUCA, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:33:38.838043785 (59h57m48.560606815s) [noise=true , address=79.127.201.215:4352 , piece_id=UQOX4NCOHWGBE4GW7DF4EJYZD5X6M55A6S47MS5EXLNUH2SDE2FQ, seg_pos={0 0}, upload=57519]: Done uploading piece. err: <nil> Jul 26 23:33:56.712565230 (59h58m6.43512827s) [noise=true , address=104.36.82.52:4087 , piece_id=NOKKGLYW73K3XPTSHVOBS34STKKC7454RBW64ILQH6JY5G2ZZR4A, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: write tcp 188.166.225.170:36248->104.36.82.52:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712612791 (59h58m6.435175831s) [noise=true , address=104.36.82.10:4095 , piece_id=SNRYXZAJDTGPUFVQXLRLYBS64USKELHLHBHGOPFC2MXHTUUXWH6Q, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:12gs9LRE3oUp1jupCw5uDvRtkNcahbq2d4VTYQsdKr6atk6vGJx, address:104.36.82.10:4095): protocol: write tcp 188.166.225.170:35886->104.36.82.10:4095: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712623201 (59h58m6.435186241s) [noise=true , address=104.36.82.46:4508 , piece_id=PWZAGWXHIXMWCMGK2B5AVWHLVAZ54DSTAOOA2JDRI5OMI2WXHMDA, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: write tcp 188.166.225.170:39846->104.36.82.46:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712656604 (59h58m6.435219645s) [noise=true , address=104.36.82.53:4156 , piece_id=IQTE2HTTO6MQQD4DBTTVMN62H3STCQVTOJZZJXYRTA4XB46JRZFQ, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 188.166.225.170:50110->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712677755 (59h58m6.435240795s) [noise=true , address=104.36.82.47:4538 , piece_id=UM23T636SQMTYUDHJJG7GW256RHFIOPQJII2UZB5X5DMLUTAADAQ, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712690750 (59h58m6.43525378s) [noise=true , address=104.36.82.54:4163 , piece_id=SJNMNVYSMYVHS47YGQSBED7N2FBZYLKBOTW6XOZV2XFJVT7ODEJA, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: write tcp 188.166.225.170:57826->104.36.82.54:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712717210 (59h58m6.43528024s) [noise=true , address=104.36.82.11:4131 , piece_id=SMM5OA5FTW55P5IBDO7K3LZFSBWIL3RSVEWOMOOH7EOR5OJI5KHA, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 188.166.225.170:38374->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712724525 (59h58m6.435287565s) [noise=true , address=104.36.82.55:4227 , piece_id=QNY42STGBS2QSRHT6IOE7OAI7CJFSDLIC2C2HCN3M4KSANETOA6A, seg_pos={0 0}, upload=57519]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: write tcp 188.166.225.170:37524->104.36.82.55:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:56.712745524 (59h58m6.435308564s) [seg_pos={0 0}, upload=57519]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:33:56.937472635 (59h58m6.660035675s) [upload=57519]: Done uploading segments... Jul 26 23:33:56.937502161 (59h58m6.660065201s) [upload=57519]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.226584598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [....X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234949944s FAIL (2025-07-26 23:33:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== ss=104.36.82.16:4293 , piece_id=GWTP7XYKW2Z6HH7TVUWVE5K2SXBAEKVQCF76JPPDYEAX7J6ZQKTQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.973177603 (59h57m35.695740633s) [noise=true , address=104.36.82.15:4259 , piece_id=R2RVFKM3XKWXYCVTVGSTWOBHWI2JA3VVK7O5VBZM5EP4CMIJEN4A, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.973197000 (59h57m35.69576004s) [noise=true , address=104.36.82.2:4729 , piece_id=C3BAYHSPPDOYGQGP4KPMHBBYW2URI652MEI7OLA77UMHWOB7DBEA, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.973473930 (59h57m35.69603697s) [noise=true , address=77.90.31.19:4842 , piece_id=5D373M4ICCIL4VCAF2IQ2B526T5VMVHCYLRZV37JLTDNWSRLO5CA, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.973530949 (59h57m35.696094039s) [noise=true , address=104.36.82.9:4006 , piece_id=6LUP6AHQ6XP2B4KPAKITAP2YODCNGPGES2POS2VJEV6K7B5KJ6QQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.973769926 (59h57m35.696332966s) [noise=true , address=77.90.31.14:4662 , piece_id=GL6AXVAP5GHV7FUHQYQMUJJHKE6TGXBJMHBGHUXMZCGEYEBVNYZQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.974188607 (59h57m35.696751687s) [noise=true , address=104.36.82.9:4005 , piece_id=GLBKLOER3MAM4KCAQPW4RF3IFOGCJYS3KLZVA4KB4DQAVYYG6VPQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.974604403 (59h57m35.697167443s) [noise=true , address=104.36.82.5:4838 , piece_id=NWL2766M67OI7JVGNLC2HF53JHJOSEXXEA3ETWBHFHR7VZQCKDPA, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.974808703 (59h57m35.697371743s) [noise=true , address=45.91.116.13:4154 , piece_id=J3U43RRIGTZZI6S34JJSENDRUI56CFZQHGDY5L47GRQU5ZERJGYQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.975131892 (59h57m35.697694932s) [noise=true , address=104.36.82.7:4910 , piece_id=CRLRIBITSJSG7MHPKQHNXSPG3SHNGBOMFQVPL25MVOJVNGB3G32A, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.975608765 (59h57m35.698171805s) [noise=true , address=104.36.82.8:4972 , piece_id=JGFPJFDYML3W7FELAGIX4LFXXNMBX2OMWXMVOZCH5ZEKRX6CZ4QA, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.976339843 (59h57m35.698902883s) [noise=true , address=104.36.82.4:4801 , piece_id=FGEAKFQUIPD7PGJQP2EJF6TPVH2BGOFU4KZB75GSUUILK5UKBNCQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.976972604 (59h57m35.699535644s) [noise=true , address=104.36.82.6:4873 , piece_id=PIKT5MV3RNZQ2M657MVWJ5U5TL2L3WPBCLXKOE473XL4IC7STYEA, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.977336840 (59h57m35.69989987s) [noise=true , address=104.36.82.3:4765 , piece_id=EJIZAAMQESEPKDTQ7EBA6PVDO2BXHFVTGKR5OE7UXOZL24QJWCAQ, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:25.977923944 (59h57m35.700486984s) [noise=true , address=104.36.82.11:4115 , piece_id=CIHOPTN4S5CIVVPCXVLH2GWBVC7RE6QU7TY7W2TZP6IXD2VR2TQA, seg_pos={0 0}, upload=57523]: Uploading piece... Jul 26 23:33:27.724595662 (59h57m37.447158692s) [upload=57523]: Next returned nil segment Jul 26 23:33:27.724614218 (59h57m37.447177258s) [upload=57523]: Waiting for error group managing segments... Jul 26 23:33:28.001743429 (59h57m37.72430655s) [noise=true , address=45.91.116.13:4154 , piece_id=J3U43RRIGTZZI6S34JJSENDRUI56CFZQHGDY5L47GRQU5ZERJGYQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: <nil> Jul 26 23:33:28.138028494 (59h57m37.860591534s) [noise=true , address=77.90.31.14:4662 , piece_id=GL6AXVAP5GHV7FUHQYQMUJJHKE6TGXBJMHBGHUXMZCGEYEBVNYZQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: <nil> Jul 26 23:33:29.488971356 (59h57m39.211534396s) [noise=true , address=77.90.31.19:4842 , piece_id=5D373M4ICCIL4VCAF2IQ2B526T5VMVHCYLRZV37JLTDNWSRLO5CA, seg_pos={0 0}, upload=57523]: Done uploading piece. err: <nil> Jul 26 23:33:55.064934935 (59h58m4.787497975s) [noise=true , address=104.36.82.13:4187 , piece_id=NCKOLELKYVVROI5RGYFJHJLHLVTITXGMOZTO72ISEEXUTEHKCSSQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.064976104 (59h58m4.787539144s) [noise=true , address=104.36.82.7:4910 , piece_id=CRLRIBITSJSG7MHPKQHNXSPG3SHNGBOMFQVPL25MVOJVNGB3G32A, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065007434 (59h58m4.787570474s) [noise=true , address=104.36.82.16:4293 , piece_id=GWTP7XYKW2Z6HH7TVUWVE5K2SXBAEKVQCF76JPPDYEAX7J6ZQKTQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065037111 (59h58m4.787600141s) [noise=true , address=104.36.82.2:4729 , piece_id=C3BAYHSPPDOYGQGP4KPMHBBYW2URI652MEI7OLA77UMHWOB7DBEA, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065069012 (59h58m4.787632052s) [noise=true , address=104.36.82.4:4801 , piece_id=FGEAKFQUIPD7PGJQP2EJF6TPVH2BGOFU4KZB75GSUUILK5UKBNCQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065097125 (59h58m4.787660166s) [noise=true , address=104.36.82.6:4873 , piece_id=PIKT5MV3RNZQ2M657MVWJ5U5TL2L3WPBCLXKOE473XL4IC7STYEA, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: rpc: dial tcp 104.36.82.6:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065130289 (59h58m4.787693329s) [noise=true , address=104.36.82.3:4765 , piece_id=EJIZAAMQESEPKDTQ7EBA6PVDO2BXHFVTGKR5OE7UXOZL24QJWCAQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:37468->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065162251 (59h58m4.787725291s) [noise=true , address=104.36.82.15:4259 , piece_id=R2RVFKM3XKWXYCVTVGSTWOBHWI2JA3VVK7O5VBZM5EP4CMIJEN4A, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065185314 (59h58m4.787748354s) [noise=true , address=104.36.82.5:4838 , piece_id=NWL2766M67OI7JVGNLC2HF53JHJOSEXXEA3ETWBHFHR7VZQCKDPA, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065232995 (59h58m4.787796036s) [noise=true , address=104.36.82.11:4115 , piece_id=CIHOPTN4S5CIVVPCXVLH2GWBVC7RE6QU7TY7W2TZP6IXD2VR2TQA, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065245390 (59h58m4.78780848s) [noise=true , address=104.36.82.9:4005 , piece_id=GLBKLOER3MAM4KCAQPW4RF3IFOGCJYS3KLZVA4KB4DQAVYYG6VPQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065266449 (59h58m4.787829489s) [noise=true , address=104.36.82.9:4006 , piece_id=6LUP6AHQ6XP2B4KPAKITAP2YODCNGPGES2POS2VJEV6K7B5KJ6QQ, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065306547 (59h58m4.787869587s) [noise=true , address=104.36.82.8:4972 , piece_id=JGFPJFDYML3W7FELAGIX4LFXXNMBX2OMWXMVOZCH5ZEKRX6CZ4QA, seg_pos={0 0}, upload=57523]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:55.065346883 (59h58m4.787909923s) [seg_pos={0 0}, upload=57523]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 26 23:33:55.297775509 (59h58m5.020338559s) [upload=57523]: Done uploading segments... Jul 26 23:33:55.297826657 (59h58m5.020389697s) [upload=57523]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.234972217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.018568371s FAIL (2025-07-26 23:33:47 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 2.018595585s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 21 out of 60 is failed (threshold=3) [...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X..] duration: 14.129265279s FAIL (2025-07-26 23:33:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.129288936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.864016957s FAIL (2025-07-26 23:33:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.86404975s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.239046085s FAIL (2025-07-26 23:33:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.239073707s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [X...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X.] duration: 50.842401933s FAIL (2025-07-26 23:33:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 50.84244684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 53 out of 60 is failed (threshold=3) [X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237723232s FAIL (2025-07-26 23:32:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== : Uploading piece... Jul 26 23:31:57.342681704 (59h56m7.065244744s) [noise=true , address=79.127.205.235:4689 , piece_id=JISSTIHBZ244SY2PLPCAARM4QWIGY4OCKFDD44ZIAWZZGGSP5DXQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.343141715 (59h56m7.065704765s) [noise=true , address=77.90.31.21:4902 , piece_id=M2XNP67Q6JSMGHTBDGXUEDWGXOFDA2SRANN3ATVKHJVMZESOMBRQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346340565 (59h56m7.068903605s) [noise=true , address=77.90.31.12:4601 , piece_id=RTSRAMSSUKHR6QCAB4HCNTH4QGLM3HR4PAAA6LDI2WCOLDVADQGA, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346494079 (59h56m7.069057119s) [noise=true , address=104.36.82.46:4494 , piece_id=5H4RKG6LLFKG6YCW74MGOICDXJFBUAX2ZWVXNW6TT6EE3JNXDA4Q, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346672069 (59h56m7.069235109s) [noise=true , address=79.127.205.246:4270 , piece_id=XQVNALQ4O24R47VUYFZHXAPTWGLZJP5NGUMVYHVSHCS2ZVT6P3SQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346870759 (59h56m7.069433799s) [noise=true , address=104.36.82.56:4241 , piece_id=AOI76JZBN42YFC76HWUGCGMMB2ZCSMFVJMEND2ISMWOKQXL6DF7Q, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.346968627 (59h56m7.069531667s) [noise=true , address=156.146.43.225:4448 , piece_id=34Q6V4DXM6NSS62WEGG4OXT7CRYFXWRUVSVP4NJXUPN4NW2NCXGA, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347192304 (59h56m7.069755344s) [noise=true , address=77.90.31.17:4793 , piece_id=I7FX3KTIYNIIDBKVF772FCU44HGUKOI64PZDOKZKXFTU7BYKG37A, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347265986 (59h56m7.069829036s) [noise=true , address=79.127.220.98:4422 , piece_id=YHVHZMXXBBBMH2YDVPJHMYJKXTTVQLLZ4RUQYUOWHKZIK5AFPADQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347559707 (59h56m7.070122747s) [noise=true , address=104.36.82.47:4549 , piece_id=Z7THAJAOWRQQYIOBAKN3WQQUFZSGW7X6FRCD3IIBQPLJIDQYGW4A, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347600175 (59h56m7.070163215s) [noise=true , address=79.127.201.215:4353 , piece_id=J3C5WCMWXBBEKL3IJKDZXT4VQQPZWBJS7NDUGSTRVNIHTB5DVXZQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.347848069 (59h56m7.070411109s) [noise=true , address=104.36.82.41:4791 , piece_id=VS3QTUH4XEENVEGAJM56BMIE4MIQMY3QJS4R3PXSFMMIB3SGJJKQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.348135328 (59h56m7.070698368s) [noise=true , address=45.91.116.9:4277 , piece_id=5427THO2OLAB5EESBTNTCEAO5GOEQS2EX7HDTUDME5LBZBGHJCDQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.348479177 (59h56m7.071042217s) [noise=true , address=64.49.13.140:4508 , piece_id=MODQHDBOKHHJHRJPLOIEQQGIAGIMRYSFMXVWUGXLRYW2ME72AARQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.348791374 (59h56m7.071354414s) [noise=true , address=79.127.219.51:4212 , piece_id=6G32LZJ3I63G3V5E6HIZBXSE35XRW57I5ZQMW72QAK4COCIJDPHQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.349085086 (59h56m7.071648126s) [noise=true , address=79.127.205.245:4239 , piece_id=4IL3DJMHGT7SFMRDRRQIJQ6MHMOYROKJFSSKEXOFQDLRDQKKPASA, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:57.349275811 (59h56m7.071838851s) [noise=true , address=104.36.82.54:4180 , piece_id=PFRJU4N3MMXIZYQWTPA2F75CN3HX5SMZDJVIUMCQDY7GMIGBHFTQ, seg_pos={0 0}, upload=57500]: Uploading piece... Jul 26 23:31:58.453333466 (59h56m8.175896496s) [noise=true , address=156.146.43.225:4448 , piece_id=34Q6V4DXM6NSS62WEGG4OXT7CRYFXWRUVSVP4NJXUPN4NW2NCXGA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 26 23:31:58.457343028 (59h56m8.179906068s) [upload=57500]: Next returned nil segment Jul 26 23:31:58.457354711 (59h56m8.179917751s) [upload=57500]: Waiting for error group managing segments... Jul 26 23:31:58.759161939 (59h56m8.48172498s) [noise=true , address=79.127.205.245:4239 , piece_id=4IL3DJMHGT7SFMRDRRQIJQ6MHMOYROKJFSSKEXOFQDLRDQKKPASA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:58.771094599 (59h56m8.493657639s) [noise=true , address=79.127.205.235:4689 , piece_id=JISSTIHBZ244SY2PLPCAARM4QWIGY4OCKFDD44ZIAWZZGGSP5DXQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:58.800868877 (59h56m8.523431917s) [noise=true , address=79.127.205.246:4270 , piece_id=XQVNALQ4O24R47VUYFZHXAPTWGLZJP5NGUMVYHVSHCS2ZVT6P3SQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:59.130345965 (59h56m8.852909005s) [noise=true , address=79.127.220.98:4422 , piece_id=YHVHZMXXBBBMH2YDVPJHMYJKXTTVQLLZ4RUQYUOWHKZIK5AFPADQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:31:59.350051629 (59h56m9.072614669s) [noise=true , address=77.90.31.21:4902 , piece_id=M2XNP67Q6JSMGHTBDGXUEDWGXOFDA2SRANN3ATVKHJVMZESOMBRQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:00.663326730 (59h56m10.38588977s) [noise=true , address=79.127.219.51:4212 , piece_id=6G32LZJ3I63G3V5E6HIZBXSE35XRW57I5ZQMW72QAK4COCIJDPHQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:01.250947466 (59h56m10.973510506s) [noise=true , address=64.49.13.140:4508 , piece_id=MODQHDBOKHHJHRJPLOIEQQGIAGIMRYSFMXVWUGXLRYW2ME72AARQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:32:01.842499864 (59h56m11.565062914s) [noise=true , address=79.127.220.101:4048 , piece_id=VN65WZ3Z24O5RIIXXH4ATRHWJ2JFRGSICNWZGD6R7ZNYCJL4N5LA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:02.379906966 (59h56m12.102470006s) [noise=true , address=45.91.116.9:4277 , piece_id=5427THO2OLAB5EESBTNTCEAO5GOEQS2EX7HDTUDME5LBZBGHJCDQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:26.935346561 (59h56m36.657909651s) [noise=true , address=79.127.201.215:4353 , piece_id=J3C5WCMWXBBEKL3IJKDZXT4VQQPZWBJS7NDUGSTRVNIHTB5DVXZQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: <nil> Jul 26 23:32:56.238172780 (59h57m5.96073582s) [noise=true , address=104.36.82.46:4494 , piece_id=5H4RKG6LLFKG6YCW74MGOICDXJFBUAX2ZWVXNW6TT6EE3JNXDA4Q, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:17b8LdpavJCCFXEBj7ndUKR6sge6Ntv96cGK5TmahF3MtwTV9z, address:104.36.82.46:4494): protocol: rpc: dial tcp 104.36.82.46:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238215362 (59h57m5.960778402s) [noise=true , address=77.90.31.12:4601 , piece_id=RTSRAMSSUKHR6QCAB4HCNTH4QGLM3HR4PAAA6LDI2WCOLDVADQGA, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2, address:77.90.31.12:4601): protocol: rpc: dial tcp 77.90.31.12:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238233165 (59h57m5.960796205s) [noise=true , address=104.36.82.54:4180 , piece_id=PFRJU4N3MMXIZYQWTPA2F75CN3HX5SMZDJVIUMCQDY7GMIGBHFTQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238252192 (59h57m5.960815232s) [noise=true , address=104.36.82.56:4241 , piece_id=AOI76JZBN42YFC76HWUGCGMMB2ZCSMFVJMEND2ISMWOKQXL6DF7Q, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 188.166.225.170:47248->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238274856 (59h57m5.960838066s) [noise=true , address=104.36.82.47:4549 , piece_id=Z7THAJAOWRQQYIOBAKN3WQQUFZSGW7X6FRCD3IIBQPLJIDQYGW4A, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: rpc: dial tcp 104.36.82.47:4549: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238326414 (59h57m5.960889454s) [noise=true , address=104.36.82.41:4791 , piece_id=VS3QTUH4XEENVEGAJM56BMIE4MIQMY3QJS4R3PXSFMMIB3SGJJKQ, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:12LpFW1mfotWPqgQxyBSimChkttujx21L6rU2GRSwvqbZ7CVeLn, address:104.36.82.41:4791): protocol: write tcp 188.166.225.170:39438->104.36.82.41:4791: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:56.238347835 (59h57m5.960910876s) [noise=true , address=77.90.31.17:4793 , piece_id=I7FX3KTIYNIIDBKVF772FCU44HGUKOI64PZDOKZKXFTU7BYKG37A, seg_pos={0 0}, upload=57500]: Done uploading piece. err: ecclient: upload failed (node:1rxrH2F4LTfrK9SMsNNEuqVr4AjV1VbEYDc3veWbQf88rj2Es4, address:77.90.31.17:4793): protocol: read tcp 188.166.225.170:40560->77.90.31.17:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:32:56.238368554 (59h57m5.960931594s) [seg_pos={0 0}, upload=57500]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:32:56.474209089 (59h57m6.196772129s) [upload=57500]: Done uploading segments... Jul 26 23:32:56.474239567 (59h57m6.196802597s) [upload=57500]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.237743762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........................XXX.X..X.....XXXXX...XXX.X.....X.] duration: 8.86876745s FAIL (2025-07-26 23:32:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.868789893s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.....X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.235877175s FAIL (2025-07-26 23:32:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== : dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205331346 (59h56m31.927894386s) [seg_pos={0 0}, upload=57499]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:32:22.440397036 (59h56m32.162960066s) [upload=57499]: Done uploading segments... Jul 26 23:32:22.440419980 (59h56m32.16298301s) [upload=57499]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:32:23.313259465 (59h56m33.035822505s) [upload=57508]: Starting upload Jul 26 23:32:23.313324229 (59h56m33.035887259s) [upload=57508]: Done starting upload Jul 26 23:32:23.313412177 (59h56m33.035975257s) [upload=57508]: Uploading segments... Jul 26 23:32:23.313574407 (59h56m33.036137447s) [upload=57508]: Got next segment. Inline: false Jul 26 23:32:23.313604294 (59h56m33.036167334s) [upload=57508]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:32:23.313610226 (59h56m33.036173266s) [upload=57508]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:32:23.761321834 (59h56m33.483884864s) [seg_pos={0 0}, upload=57508]: Begin upload segment... Jul 26 23:32:23.761361420 (59h56m33.48392446s) [seg_pos={0 0}, upload=57508]: Done begin upload segment. Jul 26 23:32:23.761377391 (59h56m33.483940431s) [noise=true , address=77.90.31.12:4592 , piece_id=IQY6RS73BDSRIHH4O4JKA4BTHM2S2BQAQ5ZPOTQ4FHV2TYMR6OKQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.761388552 (59h56m33.483951592s) [noise=true , address=104.36.82.11:4113 , piece_id=4IBHWBVZKUEQ3XH35W67Z2UXPSJCNDTAQT5F4EEUXM57EHW56XQQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.761729774 (59h56m33.484292894s) [noise=true , address=45.91.116.4:4939 , piece_id=UUCMHDCUJQGX2ERLROFNPWGADS3XZLTHMSX65MIZC7ZLNM66EHOA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762254168 (59h56m33.484817208s) [noise=true , address=45.91.116.6:4973 , piece_id=Y3NW4UNL3LCPYN4A5ZISMU3AWHYVUEY7H74PESGUCPC7ERLOBX7Q, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762382954 (59h56m33.484945994s) [noise=true , address=104.36.82.6:4875 , piece_id=FNQMYIEKWGG525N7TXWHE7UV7DGOAFAK32HRCHH4D4RZVWH7NH6Q, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762526849 (59h56m33.485089889s) [noise=true , address=77.90.31.19:4843 , piece_id=4PIBA2ORYW73FITBT3WVS373REJTRQGAWPJ3RNZB3C3QI3TI6DAA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762627141 (59h56m33.485190191s) [noise=true , address=77.90.31.21:4877 , piece_id=SXPA3ETVDY6RFUASZLBY4C2URCEQ33SQXGOT2G3KDDDSX6UOMUDA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762797687 (59h56m33.485360727s) [noise=true , address=45.91.116.7:4010 , piece_id=XYFI47RUCFZLPVOVISSYY7MTUOPMGNI2BDK5JE2S25OJWLOSOD2A, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.762842854 (59h56m33.485405904s) [noise=true , address=104.36.82.5:4837 , piece_id=V7JD52W3A5JMKDNOESLPMNIHEW4OZO64KUHZABDSYPVWNNI65AJA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763061172 (59h56m33.485624212s) [noise=true , address=104.36.82.15:4259 , piece_id=V34UN7R3KBZJC6GCOPUKXBCKDPBNTFWRDRVJCCBLNF5MQ3KPHJPQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763322842 (59h56m33.485885882s) [noise=true , address=45.91.116.14:4189 , piece_id=M26C25Q5KZECJMSQMVY5KHXU4MKVTGDZQ7LXUGIPOCIKV2EROLFA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763562401 (59h56m33.486125441s) [noise=true , address=45.91.116.13:4154 , piece_id=DHQRRAX77RWMMQDUVODWPTEI7KTXOPTMWZ4QOGF776PS24EN3HUA, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763800727 (59h56m33.486363757s) [noise=true , address=104.36.82.2:4731 , piece_id=VPU4KGAJVVOJVXXJ4KSYWPA4KMRQAPSHZR3GUA5NT7MNAFA5L7UQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.763896590 (59h56m33.48645962s) [noise=true , address=45.91.116.5:4299 , piece_id=TYMR2N4K35PSGU7GV2YL5JTVOU47X2UVFTPZBKBO3DJYVVPVTJ6A, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.764115619 (59h56m33.486678659s) [noise=true , address=45.91.116.7:4011 , piece_id=BA4YCAJPOHVJBQOIK6IPN7K532F7M6O7WO5CRDX5KX6MYUOSXA5Q, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:23.764350058 (59h56m33.486913098s) [noise=true , address=77.90.31.16:4733 , piece_id=5VRUIEUVCPT2EDPWOXMN45CJAPBJXUD5C4NPJOTTVGDIKUEKSEKQ, seg_pos={0 0}, upload=57508]: Uploading piece... Jul 26 23:32:25.490129184 (59h56m35.212692224s) [upload=57508]: Next returned nil segment Jul 26 23:32:25.490145345 (59h56m35.212708385s) [upload=57508]: Waiting for error group managing segments... Jul 26 23:32:25.775412808 (59h56m35.497975848s) [noise=true , address=45.91.116.7:4011 , piece_id=BA4YCAJPOHVJBQOIK6IPN7K532F7M6O7WO5CRDX5KX6MYUOSXA5Q, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.851963552 (59h56m35.574526592s) [noise=true , address=45.91.116.7:4010 , piece_id=XYFI47RUCFZLPVOVISSYY7MTUOPMGNI2BDK5JE2S25OJWLOSOD2A, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.862929614 (59h56m35.585492654s) [noise=true , address=45.91.116.13:4154 , piece_id=DHQRRAX77RWMMQDUVODWPTEI7KTXOPTMWZ4QOGF776PS24EN3HUA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.864755447 (59h56m35.587318487s) [noise=true , address=45.91.116.4:4939 , piece_id=UUCMHDCUJQGX2ERLROFNPWGADS3XZLTHMSX65MIZC7ZLNM66EHOA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.906462595 (59h56m35.629025625s) [noise=true , address=45.91.116.14:4189 , piece_id=M26C25Q5KZECJMSQMVY5KHXU4MKVTGDZQ7LXUGIPOCIKV2EROLFA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.929984891 (59h56m35.652547931s) [noise=true , address=45.91.116.6:4973 , piece_id=Y3NW4UNL3LCPYN4A5ZISMU3AWHYVUEY7H74PESGUCPC7ERLOBX7Q, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.950169380 (59h56m35.67273242s) [noise=true , address=77.90.31.16:4733 , piece_id=5VRUIEUVCPT2EDPWOXMN45CJAPBJXUD5C4NPJOTTVGDIKUEKSEKQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:25.951688737 (59h56m35.674251767s) [noise=true , address=45.91.116.5:4299 , piece_id=TYMR2N4K35PSGU7GV2YL5JTVOU47X2UVFTPZBKBO3DJYVVPVTJ6A, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:26.078913226 (59h56m35.801476256s) [noise=true , address=77.90.31.21:4877 , piece_id=SXPA3ETVDY6RFUASZLBY4C2URCEQ33SQXGOT2G3KDDDSX6UOMUDA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:27.366812726 (59h56m37.089375756s) [noise=true , address=77.90.31.19:4843 , piece_id=4PIBA2ORYW73FITBT3WVS373REJTRQGAWPJ3RNZB3C3QI3TI6DAA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: <nil> Jul 26 23:32:52.666109045 (59h57m2.388672085s) [noise=true , address=104.36.82.5:4837 , piece_id=V7JD52W3A5JMKDNOESLPMNIHEW4OZO64KUHZABDSYPVWNNI65AJA, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666195561 (59h57m2.388758641s) [noise=true , address=104.36.82.6:4875 , piece_id=FNQMYIEKWGG525N7TXWHE7UV7DGOAFAK32HRCHH4D4RZVWH7NH6Q, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666246458 (59h57m2.388809498s) [noise=true , address=104.36.82.15:4259 , piece_id=V34UN7R3KBZJC6GCOPUKXBCKDPBNTFWRDRVJCCBLNF5MQ3KPHJPQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666284491 (59h57m2.388847531s) [noise=true , address=77.90.31.12:4592 , piece_id=IQY6RS73BDSRIHH4O4JKA4BTHM2S2BQAQ5ZPOTQ4FHV2TYMR6OKQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666293388 (59h57m2.388856428s) [noise=true , address=104.36.82.2:4731 , piece_id=VPU4KGAJVVOJVXXJ4KSYWPA4KMRQAPSHZR3GUA5NT7MNAFA5L7UQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666300751 (59h57m2.388863792s) [noise=true , address=104.36.82.11:4113 , piece_id=4IBHWBVZKUEQ3XH35W67Z2UXPSJCNDTAQT5F4EEUXM57EHW56XQQ, seg_pos={0 0}, upload=57508]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:47632->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:52.666325309 (59h57m2.388888339s) [seg_pos={0 0}, upload=57508]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:32:52.900500787 (59h57m2.623063827s) [upload=57508]: Done uploading segments... Jul 26 23:32:52.900544341 (59h57m2.623107371s) [upload=57508]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.235897484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.234367996s FAIL (2025-07-26 23:32:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.234388957s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.868340249s FAIL (2025-07-26 23:32:38 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.868367541s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..X] duration: 1m0.001463301s FAIL (2025-07-26 23:32:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001474952s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [......X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.236924701s FAIL (2025-07-26 23:32:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== tch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:31:52.845052411 (59h56m2.567615441s) [upload=57499]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:31:53.079527764 (59h56m2.802090804s) [seg_pos={0 0}, upload=57499]: Begin upload segment... Jul 26 23:31:53.079567780 (59h56m2.80213082s) [seg_pos={0 0}, upload=57499]: Done begin upload segment. Jul 26 23:31:53.079591957 (59h56m2.802154987s) [noise=true , address=45.91.116.12:4225 , piece_id=TEHDDMAKC7P2GHBJI4E34W27YZ3CRLZSZCD7B2A7FQW5MUS5SVFA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.079639598 (59h56m2.802202638s) [noise=true , address=77.90.31.17:4770 , piece_id=HJFXXZV5TX2FZZF4LFUJT526D66IYHQTEJ6YTUNVRIWV4D3DACPA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080010728 (59h56m2.802573768s) [noise=true , address=45.91.116.8:4045 , piece_id=QXULBJBY73CKXBNNRRSQWSQKRU4L2MKLGTJVTJEF3N7LMH47AQTQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080147349 (59h56m2.802710389s) [noise=true , address=104.36.82.3:4765 , piece_id=OLEWJ6R5VRNNRWVQUYX3KSG5GANPRBXW23M55OO6KSW7XQD74BRA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080305833 (59h56m2.802868873s) [noise=true , address=104.36.82.12:4150 , piece_id=YIH7HSCLB7V5OARSUKSMTREKTEBTS6S4UYEUBE7PQXBLNIW77W2A, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080573715 (59h56m2.803136755s) [noise=true , address=45.91.116.10:4082 , piece_id=JLVQYPCY6FEJJ54YPX5GHMMQTDYEOT6PIBAHL4DWXU2WH3TJ25VQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080628801 (59h56m2.803191831s) [noise=true , address=104.36.82.7:4910 , piece_id=X6I37G3J7KPYZVBOXGEADLTECWJRURRVVW5TKGRNIASOBT6M36ZQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080869512 (59h56m2.803432552s) [noise=true , address=104.36.82.6:4874 , piece_id=637BK2LTKHCYE34SMXRZSYL6S436WO7DHOFSBS43M7SPQZALTZ6A, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.080888517 (59h56m2.803451547s) [noise=true , address=104.36.82.13:4187 , piece_id=BDXQXYRHLV7TI326PPYYWHVM2VZWDWHZZHUSVTCIEUKD4KKYMBFQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081210333 (59h56m2.803773373s) [noise=true , address=104.36.82.4:4801 , piece_id=IGIYCYLGSAJXQ7JQMVJRKFNSNNX2JQ7EK6XIMJHXJCNHRBJXOHQA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081216144 (59h56m2.803779174s) [noise=true , address=77.90.31.10:4520 , piece_id=YUTK6OVNPAOJ66IYNBUZZVLMEUSBE46ZL34RKMWWZF5Z4ZNUQ2GQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081490639 (59h56m2.804053689s) [noise=true , address=45.91.116.14:4190 , piece_id=4OHJMBE3HJRPYMDDQD7MDFRLJR2PKBOXOLZYPCPREVF2JH24HGTA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081509566 (59h56m2.804072596s) [noise=true , address=45.91.116.7:4009 , piece_id=ZA6MQOJ4D74FY4QCDOAP52UAJRSJQRCHQG2IILYTE4SFJJNQQGPA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081732582 (59h56m2.804295623s) [noise=true , address=104.36.82.15:4259 , piece_id=7JPY57ZC4BQKU4HPKQIZ3NIH2TPQ735FK45M5F2SHINTTOJMLCUQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081804571 (59h56m2.804367611s) [noise=true , address=104.36.82.14:4221 , piece_id=VXFUFOELD4IUZER4R7ULGNFGWT6PBW4GNIL5S7CMCHOKUIBKXYEQ, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:53.081986508 (59h56m2.804549548s) [noise=true , address=45.91.116.13:4154 , piece_id=37EKK7P2QHRL36Q7LEWJHNVZSJHBYLDCNU2KW4BYVL2LBWLBTBAA, seg_pos={0 0}, upload=57499]: Uploading piece... Jul 26 23:31:54.825954936 (59h56m4.548517976s) [upload=57499]: Next returned nil segment Jul 26 23:31:54.825973040 (59h56m4.54853608s) [upload=57499]: Waiting for error group managing segments... Jul 26 23:31:55.103202033 (59h56m4.825765063s) [noise=true , address=45.91.116.13:4154 , piece_id=37EKK7P2QHRL36Q7LEWJHNVZSJHBYLDCNU2KW4BYVL2LBWLBTBAA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.110086025 (59h56m4.832649065s) [noise=true , address=45.91.116.10:4082 , piece_id=JLVQYPCY6FEJJ54YPX5GHMMQTDYEOT6PIBAHL4DWXU2WH3TJ25VQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.112825736 (59h56m4.835388786s) [noise=true , address=45.91.116.14:4190 , piece_id=4OHJMBE3HJRPYMDDQD7MDFRLJR2PKBOXOLZYPCPREVF2JH24HGTA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.191643428 (59h56m4.914206468s) [noise=true , address=77.90.31.10:4520 , piece_id=YUTK6OVNPAOJ66IYNBUZZVLMEUSBE46ZL34RKMWWZF5Z4ZNUQ2GQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.191826358 (59h56m4.914389388s) [noise=true , address=45.91.116.8:4045 , piece_id=QXULBJBY73CKXBNNRRSQWSQKRU4L2MKLGTJVTJEF3N7LMH47AQTQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.232874405 (59h56m4.955437445s) [noise=true , address=45.91.116.7:4009 , piece_id=ZA6MQOJ4D74FY4QCDOAP52UAJRSJQRCHQG2IILYTE4SFJJNQQGPA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:31:55.244022796 (59h56m4.966585836s) [noise=true , address=45.91.116.12:4225 , piece_id=TEHDDMAKC7P2GHBJI4E34W27YZ3CRLZSZCD7B2A7FQW5MUS5SVFA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: <nil> Jul 26 23:32:22.205093270 (59h56m31.92765631s) [noise=true , address=104.36.82.12:4150 , piece_id=YIH7HSCLB7V5OARSUKSMTREKTEBTS6S4UYEUBE7PQXBLNIW77W2A, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:60362->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205155129 (59h56m31.927718169s) [noise=true , address=104.36.82.15:4259 , piece_id=7JPY57ZC4BQKU4HPKQIZ3NIH2TPQ735FK45M5F2SHINTTOJMLCUQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 188.166.225.170:57980->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205199333 (59h56m31.927762373s) [noise=true , address=104.36.82.14:4221 , piece_id=VXFUFOELD4IUZER4R7ULGNFGWT6PBW4GNIL5S7CMCHOKUIBKXYEQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 188.166.225.170:53160->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205207129 (59h56m31.927770169s) [noise=true , address=77.90.31.17:4770 , piece_id=HJFXXZV5TX2FZZF4LFUJT526D66IYHQTEJ6YTUNVRIWV4D3DACPA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:44874->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:44874->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44874->77.90.31.17:4770: use of closed network connection Jul 26 23:32:22.205240272 (59h56m31.927803312s) [noise=true , address=104.36.82.6:4874 , piece_id=637BK2LTKHCYE34SMXRZSYL6S436WO7DHOFSBS43M7SPQZALTZ6A, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205256913 (59h56m31.927819953s) [noise=true , address=104.36.82.3:4765 , piece_id=OLEWJ6R5VRNNRWVQUYX3KSG5GANPRBXW23M55OO6KSW7XQD74BRA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205285999 (59h56m31.927849039s) [noise=true , address=104.36.82.13:4187 , piece_id=BDXQXYRHLV7TI326PPYYWHVM2VZWDWHZZHUSVTCIEUKD4KKYMBFQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205290939 (59h56m31.927853989s) [noise=true , address=104.36.82.7:4910 , piece_id=X6I37G3J7KPYZVBOXGEADLTECWJRURRVVW5TKGRNIASOBT6M36ZQ, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205296179 (59h56m31.927859209s) [noise=true , address=104.36.82.4:4801 , piece_id=IGIYCYLGSAJXQ7JQMVJRKFNSNNX2JQ7EK6XIMJHXJCNHRBJXOHQA, seg_pos={0 0}, upload=57499]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:22.205331346 (59h56m31.927894386s) [seg_pos={0 0}, upload=57499]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:32:22.440397036 (59h56m32.162960066s) [upload=57499]: Done uploading segments... Jul 26 23:32:22.440419980 (59h56m32.16298301s) [upload=57499]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.236948245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.2957381s FAIL (2025-07-26 23:32:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.295763851s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [............................XXX.X..X.....XXXXX...XXX.X.....X] duration: 30.000868758s FAIL (2025-07-26 23:32:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000891051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235752063s FAIL (2025-07-26 23:31:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ading piece... Jul 26 23:30:54.178707139 (59h55m3.901270169s) [noise=true , address=79.127.219.38:4003 , piece_id=OIWN4NLDYBXLKAYBVHXJVDJ5PIRB5FGE6LOJR66E2B5MWJRB7UHQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.178802241 (59h55m3.901365271s) [noise=true , address=77.90.31.10:4540 , piece_id=SAJ2UXKVDCCWVG3UUV4VYR25NOVXL2I3EANBZYUH2R2HY72WRXWQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.178960754 (59h55m3.901523795s) [noise=true , address=64.49.13.140:4514 , piece_id=SVCBCM2BZDW7D556JAJX64KAFZ3LFRY6AHZ7SDMY5PS6JVRSX5RQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179068891 (59h55m3.901631931s) [noise=true , address=104.36.82.46:4485 , piece_id=5TPDMQZPBD3ZS4PUNAXVBPKIFEF5L2LIEM4YFFHVV3DZBDJW66SQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179209039 (59h55m3.90177208s) [noise=true , address=104.36.82.15:4276 , piece_id=F6BDHCP5PH3XFG4X7FCYHLS62JYBEX36LDYQY3ISBKOAOW64SDKA, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179330291 (59h55m3.901893331s) [noise=true , address=104.36.82.47:4536 , piece_id=TTWORDXUOWWJRT5TQS36TPBEAN2KS4PK6WM5XG2VQL3FH2LHXDKQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.179555292 (59h55m3.902118362s) [noise=true , address=77.90.31.12:4611 , piece_id=SOXJOGSW2UKAHTC2XDGLE5REZAKGB2NDV6N4E7QTRBGTG3WG6TQA, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.180001546 (59h55m3.902564586s) [noise=true , address=104.36.82.53:4148 , piece_id=PT4RBZF62VD4Y7REHJQZWXRO4LB4EAN43FLNDG5AQAEJ3NLIQXUA, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.180532051 (59h55m3.903095101s) [noise=true , address=64.49.13.138:4482 , piece_id=2R36EPQ6IEDQHY3EE4VOM2KETVRKYKM75XN75X42GPB6Y77OZSEQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.180786066 (59h55m3.903349096s) [noise=true , address=104.36.82.18:4390 , piece_id=DUHRIT3GPWAUEVTU3L5BCQNKUFLRLAT6H5VNJTJ3NR2O3SBV7AGQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181068067 (59h55m3.903631107s) [noise=true , address=77.90.31.11:4558 , piece_id=SM24PRN6R74SSJHWVERKYEY4G5FTLE7RCUH2L3NSNM4J6S6O5JSQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181375244 (59h55m3.903938284s) [noise=true , address=104.36.82.55:4195 , piece_id=BZNQWSAKOTQVOLB6TZYBY4OZ3GPRZV2R5VQP33ENTYHNAVKAWJCQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181595716 (59h55m3.904158756s) [noise=true , address=104.36.82.52:4118 , piece_id=JNVDAPVCYI7DWVWBTTRR6UI5NUSI7BLHZ5L3M6RUPGKVW4DBRGDQ, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:54.181859280 (59h55m3.90442232s) [noise=true , address=104.36.82.11:4125 , piece_id=KQJ55Y34PKKXTHG7XNCAXNXSGATNZCMJRZY2B22RX342KLREDX6Q, seg_pos={0 0}, upload=57488]: Uploading piece... Jul 26 23:30:55.285566215 (59h55m5.008129255s) [upload=57488]: Next returned nil segment Jul 26 23:30:55.285584069 (59h55m5.008147109s) [upload=57488]: Waiting for error group managing segments... Jul 26 23:30:55.597217568 (59h55m5.319780608s) [noise=true , address=79.127.205.245:4245 , piece_id=HMSYQQFNUJZAWOOCH4FDKSFRHA5WBBDV2HEDIAJGXU7BGDNOT72Q, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:55.618511900 (59h55m5.34107494s) [noise=true , address=79.127.205.233:4655 , piece_id=TZHKBUBWCE3JBYSZ6LGMLMVUH5P7B37Z4VN6U4XPZQBQB5X62XGA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:55.709907568 (59h55m5.432470599s) [noise=true , address=79.127.205.246:4282 , piece_id=AIZOQPN6BTCGUVNXL3ZO27XWLTNIHCWWX7NHMK3LNERE4LKRFTYQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:56.054055077 (59h55m5.776618117s) [noise=true , address=79.127.219.38:4003 , piece_id=OIWN4NLDYBXLKAYBVHXJVDJ5PIRB5FGE6LOJR66E2B5MWJRB7UHQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:56.159473544 (59h55m5.882036584s) [noise=true , address=77.90.31.10:4540 , piece_id=SAJ2UXKVDCCWVG3UUV4VYR25NOVXL2I3EANBZYUH2R2HY72WRXWQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:30:57.266956595 (59h55m6.989519635s) [noise=true , address=64.49.13.138:4482 , piece_id=2R36EPQ6IEDQHY3EE4VOM2KETVRKYKM75XN75X42GPB6Y77OZSEQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:31:04.113925109 (59h55m13.836488149s) [noise=true , address=64.49.13.140:4514 , piece_id=SVCBCM2BZDW7D556JAJX64KAFZ3LFRY6AHZ7SDMY5PS6JVRSX5RQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:31:49.297577029 (59h55m59.020140069s) [noise=true , address=79.127.201.215:4349 , piece_id=QLWJ4NIMF4XOUM5YKNIDA2DVME6CNIRZ27KTTH4Z7RMJA52VHJOQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: <nil> Jul 26 23:31:53.064761292 (59h56m2.787324322s) [noise=true , address=77.90.31.12:4611 , piece_id=SOXJOGSW2UKAHTC2XDGLE5REZAKGB2NDV6N4E7QTRBGTG3WG6TQA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: write tcp 188.166.225.170:54474->77.90.31.12:4611: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064807680 (59h56m2.78737072s) [noise=true , address=104.36.82.18:4390 , piece_id=DUHRIT3GPWAUEVTU3L5BCQNKUFLRLAT6H5VNJTJ3NR2O3SBV7AGQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: write tcp 188.166.225.170:50360->104.36.82.18:4390: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064838248 (59h56m2.787401278s) [noise=true , address=104.36.82.46:4485 , piece_id=5TPDMQZPBD3ZS4PUNAXVBPKIFEF5L2LIEM4YFFHVV3DZBDJW66SQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: write tcp 188.166.225.170:37788->104.36.82.46:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064871302 (59h56m2.787434342s) [noise=true , address=104.36.82.15:4276 , piece_id=F6BDHCP5PH3XFG4X7FCYHLS62JYBEX36LDYQY3ISBKOAOW64SDKA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: write tcp 188.166.225.170:39702->104.36.82.15:4276: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.064992564 (59h56m2.787555594s) [noise=true , address=104.36.82.11:4125 , piece_id=KQJ55Y34PKKXTHG7XNCAXNXSGATNZCMJRZY2B22RX342KLREDX6Q, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: write tcp 188.166.225.170:47236->104.36.82.11:4125: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065023192 (59h56m2.787586232s) [noise=true , address=104.36.82.53:4148 , piece_id=PT4RBZF62VD4Y7REHJQZWXRO4LB4EAN43FLNDG5AQAEJ3NLIQXUA, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: write tcp 188.166.225.170:60288->104.36.82.53:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065028081 (59h56m2.787591121s) [noise=true , address=104.36.82.52:4118 , piece_id=JNVDAPVCYI7DWVWBTTRR6UI5NUSI7BLHZ5L3M6RUPGKVW4DBRGDQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: write tcp 188.166.225.170:42958->104.36.82.52:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065032661 (59h56m2.787595691s) [noise=true , address=77.90.31.11:4558 , piece_id=SM24PRN6R74SSJHWVERKYEY4G5FTLE7RCUH2L3NSNM4J6S6O5JSQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:60078->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065085050 (59h56m2.78764809s) [noise=true , address=104.36.82.55:4195 , piece_id=BZNQWSAKOTQVOLB6TZYBY4OZ3GPRZV2R5VQP33ENTYHNAVKAWJCQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: write tcp 188.166.225.170:32814->104.36.82.55:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065091874 (59h56m2.787654904s) [noise=true , address=104.36.82.47:4536 , piece_id=TTWORDXUOWWJRT5TQS36TPBEAN2KS4PK6WM5XG2VQL3FH2LHXDKQ, seg_pos={0 0}, upload=57488]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 188.166.225.170:37538->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.065103396 (59h56m2.787666436s) [seg_pos={0 0}, upload=57488]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:31:53.297872715 (59h56m3.020435755s) [upload=57488]: Done uploading segments... Jul 26 23:31:53.297901179 (59h56m3.020464219s) [upload=57488]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.23576588s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.............................XXX.X..X.....XXXXX...XXX.X.....] duration: 7.59956393s FAIL (2025-07-26 23:31:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.599577676s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [.XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX..] duration: 35.883120325s FAIL (2025-07-26 23:31:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 35.883132749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.853814106s FAIL (2025-07-26 23:31:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.853846298s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.......X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.237683038s FAIL (2025-07-26 23:31:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 67]: Done uploading segments... Jul 26 23:30:00.101794078 (59h54m9.824357118s) [upload=57467]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 26 23:30:40.084306704 (59h54m49.806869744s) [upload=57482]: Starting upload Jul 26 23:30:40.084364915 (59h54m49.806927945s) [upload=57482]: Done starting upload Jul 26 23:30:40.084446151 (59h54m49.807009191s) [upload=57482]: Uploading segments... Jul 26 23:30:40.084637487 (59h54m49.807200527s) [upload=57482]: Got next segment. Inline: false Jul 26 23:30:40.084668797 (59h54m49.807231827s) [upload=57482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:30:40.084673335 (59h54m49.807236375s) [upload=57482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:30:40.531458710 (59h54m50.254021751s) [seg_pos={0 0}, upload=57482]: Begin upload segment... Jul 26 23:30:40.531557320 (59h54m50.25412036s) [seg_pos={0 0}, upload=57482]: Done begin upload segment. Jul 26 23:30:40.531581936 (59h54m50.254144976s) [noise=true , address=77.90.31.18:4807 , piece_id=FMQJOAME6X4LLR3DCTAWKRKVZJVJQZFZ6DTLZC4CFIBBVJKOSWOQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.531590152 (59h54m50.254153192s) [noise=true , address=104.36.82.2:4729 , piece_id=R5SGBXXZLH64BCHB3NCJKJUXPP4UBQ5FPU4O5OVZA7EWM7NLIVMA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532114545 (59h54m50.254677585s) [noise=true , address=45.91.116.13:4154 , piece_id=37LBS2PPJKGVSWSAMD6PBZSQ6QCYDU6E7PWPPJOEJL6P2IH4LW4A, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532330970 (59h54m50.25489403s) [noise=true , address=104.36.82.7:4912 , piece_id=4CJ7X2Q3FTLXXDY3NMWABVPMCTMY5W6HSGPWDU6PQDSXTT265OSA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532564817 (59h54m50.255127857s) [noise=true , address=45.91.116.4:4939 , piece_id=YNUMY7RKUV64QPQEYPLGJICZL42GP55ANLPWJKV4MO6CKWZP4MZQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.532979841 (59h54m50.255542901s) [noise=true , address=104.36.82.10:4079 , piece_id=JE4DDTPWHGGAEUFUDU2N6AQUF7LRIP3S3GG6SDUJBZ6A42RGPIWA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.533980686 (59h54m50.256543726s) [noise=true , address=104.36.82.9:4006 , piece_id=5SF5SAJMBPVD4JR2IFPV323POFHW6VWHI2CBRBT6L5ZIGLKLAX7Q, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534250111 (59h54m50.256813151s) [noise=true , address=77.90.31.19:4842 , piece_id=C4TTNABYCSPPF2MWHSIHTU56GON4ZS45QZJLGNCHV6HHFGCZCMEA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534464070 (59h54m50.25702711s) [noise=true , address=104.36.82.16:4295 , piece_id=DQSPK6R3VZSPJZSHLP7WM2GDAUYX6MUBVCDUFKVDMOVZ2MPSYA3Q, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534684402 (59h54m50.257247432s) [noise=true , address=104.36.82.7:4911 , piece_id=RPPWFAMZIDXXA62II77JSRGZXFSZE3IHWYBDYGKD7S5AJWGRQQ4A, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.534898221 (59h54m50.257461261s) [noise=true , address=45.91.116.6:4975 , piece_id=TKKDKPFZY432JZ4QJYZKTW3HYKPI43V7XGDEBOMWRPXWYKE2HRUA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535097552 (59h54m50.257660592s) [noise=true , address=45.91.116.12:4227 , piece_id=K3PGMRXOQQZ53HHZJJXXBDUCYMXC4NCRK3SLJRWYLXCSDHSM67XQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535286404 (59h54m50.257849444s) [noise=true , address=45.91.116.14:4189 , piece_id=3WZXZVVWQ4CCNZGS2MLWGF5J2XUEYF35UJ2KAUV63FHWARTNZAVQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535481337 (59h54m50.258044367s) [noise=true , address=45.91.116.5:4297 , piece_id=BIZYKQK7FDCWSCLVM2T7VYZ7VZWOXDJXBV7ALQFOGGSHECHC4BWA, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535719583 (59h54m50.258282623s) [noise=true , address=77.90.31.14:4661 , piece_id=NWG65BD3VTJFJIJKGMDWZJ3LLYQNLXMCCU3FXNOXNEYRBXZZYJIQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:40.535970052 (59h54m50.258533092s) [noise=true , address=77.90.31.21:4877 , piece_id=FAF25QKKR2BAW5TXDTLLO5FQVLBBIONTSKMIYPIX4ZCQCWKTBEKQ, seg_pos={0 0}, upload=57482]: Uploading piece... Jul 26 23:30:42.267284129 (59h54m51.989847169s) [upload=57482]: Next returned nil segment Jul 26 23:30:42.267330007 (59h54m51.989893047s) [upload=57482]: Waiting for error group managing segments... Jul 26 23:30:42.604913796 (59h54m52.327476866s) [noise=true , address=45.91.116.14:4189 , piece_id=3WZXZVVWQ4CCNZGS2MLWGF5J2XUEYF35UJ2KAUV63FHWARTNZAVQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.611863303 (59h54m52.334426333s) [noise=true , address=45.91.116.4:4939 , piece_id=YNUMY7RKUV64QPQEYPLGJICZL42GP55ANLPWJKV4MO6CKWZP4MZQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.630485513 (59h54m52.353048553s) [noise=true , address=77.90.31.21:4877 , piece_id=FAF25QKKR2BAW5TXDTLLO5FQVLBBIONTSKMIYPIX4ZCQCWKTBEKQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.653457135 (59h54m52.376020175s) [noise=true , address=45.91.116.6:4975 , piece_id=TKKDKPFZY432JZ4QJYZKTW3HYKPI43V7XGDEBOMWRPXWYKE2HRUA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.703353371 (59h54m52.425916411s) [noise=true , address=45.91.116.12:4227 , piece_id=K3PGMRXOQQZ53HHZJJXXBDUCYMXC4NCRK3SLJRWYLXCSDHSM67XQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.704483914 (59h54m52.427046944s) [noise=true , address=45.91.116.13:4154 , piece_id=37LBS2PPJKGVSWSAMD6PBZSQ6QCYDU6E7PWPPJOEJL6P2IH4LW4A, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:42.712981372 (59h54m52.435544412s) [noise=true , address=77.90.31.14:4661 , piece_id=NWG65BD3VTJFJIJKGMDWZJ3LLYQNLXMCCU3FXNOXNEYRBXZZYJIQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:43.076229695 (59h54m52.798792725s) [noise=true , address=45.91.116.5:4297 , piece_id=BIZYKQK7FDCWSCLVM2T7VYZ7VZWOXDJXBV7ALQFOGGSHECHC4BWA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:30:44.199539605 (59h54m53.922102645s) [noise=true , address=77.90.31.19:4842 , piece_id=C4TTNABYCSPPF2MWHSIHTU56GON4ZS45QZJLGNCHV6HHFGCZCMEA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: <nil> Jul 26 23:31:09.442833821 (59h55m19.165396851s) [noise=true , address=104.36.82.16:4295 , piece_id=DQSPK6R3VZSPJZSHLP7WM2GDAUYX6MUBVCDUFKVDMOVZ2MPSYA3Q, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442901150 (59h55m19.16546419s) [noise=true , address=77.90.31.18:4807 , piece_id=FMQJOAME6X4LLR3DCTAWKRKVZJVJQZFZ6DTLZC4CFIBBVJKOSWOQ, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:35858->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442952097 (59h55m19.165515137s) [noise=true , address=104.36.82.10:4079 , piece_id=JE4DDTPWHGGAEUFUDU2N6AQUF7LRIP3S3GG6SDUJBZ6A42RGPIWA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:55318->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442964661 (59h55m19.165527701s) [noise=true , address=104.36.82.7:4911 , piece_id=RPPWFAMZIDXXA62II77JSRGZXFSZE3IHWYBDYGKD7S5AJWGRQQ4A, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442976423 (59h55m19.165539464s) [noise=true , address=104.36.82.2:4729 , piece_id=R5SGBXXZLH64BCHB3NCJKJUXPP4UBQ5FPU4O5OVZA7EWM7NLIVMA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.442996783 (59h55m19.165559823s) [noise=true , address=104.36.82.9:4006 , piece_id=5SF5SAJMBPVD4JR2IFPV323POFHW6VWHI2CBRBT6L5ZIGLKLAX7Q, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.443012263 (59h55m19.165575313s) [noise=true , address=104.36.82.7:4912 , piece_id=4CJ7X2Q3FTLXXDY3NMWABVPMCTMY5W6HSGPWDU6PQDSXTT265OSA, seg_pos={0 0}, upload=57482]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 188.166.225.170:54992->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:09.443061656 (59h55m19.165624696s) [seg_pos={0 0}, upload=57482]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:31:09.677997009 (59h55m19.400560049s) [upload=57482]: Done uploading segments... Jul 26 23:31:09.678047015 (59h55m19.400610065s) [upload=57482]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.237714849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.241701548s FAIL (2025-07-26 23:31:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.241730314s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 22 out of 60 is failed (threshold=3) [..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX.] duration: 25.382920057s FAIL (2025-07-26 23:30:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.38293201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [X.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23196722s FAIL (2025-07-26 23:30:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 3:29:53.250824302 (59h54m2.973387342s) [upload=57474]: Uploading segments... Jul 26 23:29:53.250999246 (59h54m2.973562287s) [upload=57474]: Got next segment. Inline: false Jul 26 23:29:53.251052939 (59h54m2.973615979s) [upload=57474]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:53.251057909 (59h54m2.973620949s) [upload=57474]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:53.707199345 (59h54m3.429762385s) [seg_pos={0 0}, upload=57474]: Begin upload segment... Jul 26 23:29:53.707275330 (59h54m3.42983837s) [seg_pos={0 0}, upload=57474]: Done begin upload segment. Jul 26 23:29:53.707320868 (59h54m3.429883948s) [noise=true , address=79.127.201.215:4351 , piece_id=E3GQBT6MVC2YLLYDLB6BBIUA2SGINRPIFY7U54H63QRELIIFRRUQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.707713709 (59h54m3.430276809s) [noise=true , address=104.36.82.52:4111 , piece_id=NN6VQF6RNUXXXP6YJEO3U24WC33PHEDQ7M66CYQ4G7YEFJ33R4EQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708100429 (59h54m3.430663469s) [noise=true , address=104.36.82.54:4165 , piece_id=XHXFX7MUDXNVB6BGMXWCZ6YPOSQ5ZH7EHNYKLOJZIAPVMMJEF6UQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708433466 (59h54m3.430996506s) [noise=true , address=104.36.82.46:4515 , piece_id=RMODAKQZ6OUF2N2AN5DTYL5AP3RE2PGC46VEWJLR4O4BPLLIJPKQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708621326 (59h54m3.431184386s) [noise=true , address=79.127.205.245:4249 , piece_id=TLSRGN5GMGUUK6X3EKE4ZWSDBEKMHXM5UJNYLFCDT4Z4WHZ2KJTA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.708991523 (59h54m3.431554574s) [noise=true , address=156.146.43.225:4434 , piece_id=2YRK4ZD2CJ4WRVASOPOFE6J2NGEVXNC3YOHE2ZTJ635WHZBFDQPQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709039015 (59h54m3.431602055s) [noise=true , address=79.127.205.235:4673 , piece_id=MLVD3LC32RVPS5RRXUGJNDSGKGHKA2UJYDTKF3NFLVG7JR2CBOZQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709422939 (59h54m3.431985979s) [noise=true , address=79.127.205.234:4601 , piece_id=QNNWJUEHWX4VKYDK32LJCLEWEDCQCN4QP7343FO5MYWUO26KGOXQ, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709466283 (59h54m3.432029313s) [noise=true , address=79.127.205.246:4272 , piece_id=AUEXTAZP7E2WEU223JAUVWBAQ37PFUKF4PLQQ6R6C3VBP3WGOPZA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709849045 (59h54m3.432412105s) [noise=true , address=104.36.82.47:4538 , piece_id=MU2IOTANZ55QDD25IEZ7CSO5HOSRQE7DOAXOOVP5KDMDXAZGS2FA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.709928297 (59h54m3.432491337s) [noise=true , address=45.91.116.5:4307 , piece_id=4WS5QO6M4RENFGTOBNDL7BNJIKVBFBUOP7F4I6H4V5CO2JKANX4A, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710199585 (59h54m3.432762635s) [noise=true , address=45.91.116.14:4206 , piece_id=3E6F7BFBUAMD4EQ5FIZ46T5TIPWFUY4H6Z5NLPHDHUAZRVFTAWOA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710339664 (59h54m3.432902714s) [noise=true , address=45.91.116.14:4222 , piece_id=62DMQYPYQAQE4OWS7OB22QRHRHGHAXRE4VWZXTGBJCWAW4F2S2GA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710503407 (59h54m3.433066447s) [noise=true , address=77.90.31.17:4792 , piece_id=CKDNLHTAZDFEIOB623NRG3DN227ND3MTDDWJ5XS5DQ23SFQPS2TA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710686807 (59h54m3.433249858s) [noise=true , address=77.90.31.13:4639 , piece_id=CANLYTCYNGZVGDKGD4EK5XJJCBFAOIB2SYP2UBUEYRPJBTQJ3UWA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.710809413 (59h54m3.433372453s) [noise=true , address=79.127.220.97:4361 , piece_id=GVE3D45YX732VKFTQH265VYH6PMLNLK6R3OZZ2YBHTKBJ5KNFRIA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.711013202 (59h54m3.433576252s) [noise=true , address=156.146.43.225:4449 , piece_id=SEREWESX23R2IRYHSXMTOTMIBHEGNVWFV3E3VVDB43X2RWHRQFRA, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:53.711343704 (59h54m3.433906744s) [noise=true , address=64.49.13.138:4483 , piece_id=OE5HU663BHCSNERNU6GW3LV7G5OTWNJNV6YEGZCOUWCCSNSNOE4Q, seg_pos={0 0}, upload=57474]: Uploading piece... Jul 26 23:29:54.807016598 (59h54m4.529579628s) [upload=57474]: Next returned nil segment Jul 26 23:29:54.807036777 (59h54m4.529599817s) [upload=57474]: Waiting for error group managing segments... Jul 26 23:29:55.131554306 (59h54m4.854117336s) [noise=true , address=79.127.205.245:4249 , piece_id=TLSRGN5GMGUUK6X3EKE4ZWSDBEKMHXM5UJNYLFCDT4Z4WHZ2KJTA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.143541201 (59h54m4.866104241s) [noise=true , address=79.127.205.234:4601 , piece_id=QNNWJUEHWX4VKYDK32LJCLEWEDCQCN4QP7343FO5MYWUO26KGOXQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.216734031 (59h54m4.939297061s) [noise=true , address=79.127.205.246:4272 , piece_id=AUEXTAZP7E2WEU223JAUVWBAQ37PFUKF4PLQQ6R6C3VBP3WGOPZA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.308789230 (59h54m5.03135229s) [noise=true , address=79.127.205.235:4673 , piece_id=MLVD3LC32RVPS5RRXUGJNDSGKGHKA2UJYDTKF3NFLVG7JR2CBOZQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.691662282 (59h54m5.414225322s) [noise=true , address=45.91.116.14:4222 , piece_id=62DMQYPYQAQE4OWS7OB22QRHRHGHAXRE4VWZXTGBJCWAW4F2S2GA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:55.754482246 (59h54m5.477045286s) [noise=true , address=77.90.31.17:4792 , piece_id=CKDNLHTAZDFEIOB623NRG3DN227ND3MTDDWJ5XS5DQ23SFQPS2TA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:56.252467814 (59h54m5.975030844s) [noise=true , address=45.91.116.5:4307 , piece_id=4WS5QO6M4RENFGTOBNDL7BNJIKVBFBUOP7F4I6H4V5CO2JKANX4A, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:56.616698146 (59h54m6.339261176s) [noise=true , address=64.49.13.138:4483 , piece_id=OE5HU663BHCSNERNU6GW3LV7G5OTWNJNV6YEGZCOUWCCSNSNOE4Q, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:57.234672286 (59h54m6.957235386s) [noise=true , address=156.146.43.225:4434 , piece_id=2YRK4ZD2CJ4WRVASOPOFE6J2NGEVXNC3YOHE2ZTJ635WHZBFDQPQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:57.508445900 (59h54m7.23100894s) [noise=true , address=45.91.116.14:4206 , piece_id=3E6F7BFBUAMD4EQ5FIZ46T5TIPWFUY4H6Z5NLPHDHUAZRVFTAWOA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:57.678401365 (59h54m7.400964395s) [noise=true , address=156.146.43.225:4449 , piece_id=SEREWESX23R2IRYHSXMTOTMIBHEGNVWFV3E3VVDB43X2RWHRQFRA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:29:58.456154200 (59h54m8.17871723s) [noise=true , address=79.127.220.97:4361 , piece_id=GVE3D45YX732VKFTQH265VYH6PMLNLK6R3OZZ2YBHTKBJ5KNFRIA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:30:31.761179840 (59h54m41.483742911s) [noise=true , address=79.127.201.215:4351 , piece_id=E3GQBT6MVC2YLLYDLB6BBIUA2SGINRPIFY7U54H63QRELIIFRRUQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: <nil> Jul 26 23:30:52.604814246 (59h55m2.327377286s) [noise=true , address=77.90.31.13:4639 , piece_id=CANLYTCYNGZVGDKGD4EK5XJJCBFAOIB2SYP2UBUEYRPJBTQJ3UWA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12N9N1pcaVn3nPh1WY7uktz8Z7fwJGqnTdVwZMAW5SVrqvRA8VV, address:77.90.31.13:4639): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:30:52.604921631 (59h55m2.327484671s) [noise=true , address=104.36.82.54:4165 , piece_id=XHXFX7MUDXNVB6BGMXWCZ6YPOSQ5ZH7EHNYKLOJZIAPVMMJEF6UQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: write tcp 188.166.225.170:59972->104.36.82.54:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.604958531 (59h55m2.327521571s) [noise=true , address=104.36.82.46:4515 , piece_id=RMODAKQZ6OUF2N2AN5DTYL5AP3RE2PGC46VEWJLR4O4BPLLIJPKQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 188.166.225.170:37572->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.604990633 (59h55m2.327553673s) [noise=true , address=104.36.82.52:4111 , piece_id=NN6VQF6RNUXXXP6YJEO3U24WC33PHEDQ7M66CYQ4G7YEFJ33R4EQ, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 188.166.225.170:35610->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.605016552 (59h55m2.327579602s) [noise=true , address=104.36.82.47:4538 , piece_id=MU2IOTANZ55QDD25IEZ7CSO5HOSRQE7DOAXOOVP5KDMDXAZGS2FA, seg_pos={0 0}, upload=57474]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: write tcp 188.166.225.170:46714->104.36.82.47:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:52.605051108 (59h55m2.327614148s) [seg_pos={0 0}, upload=57474]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:30:52.833915595 (59h55m2.556478635s) [upload=57474]: Done uploading segments... Jul 26 23:30:52.833941204 (59h55m2.556504244s) [upload=57474]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.231982218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.894924019s FAIL (2025-07-26 23:30:30 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.894949288s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 23 out of 60 is failed (threshold=3) [X..XX...X.X.X.X.......XX..XX.XX..XXX............XXXX.....XXX] duration: 1m0.000557355s FAIL (2025-07-26 23:30:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000580559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................XXX.X..X.....XXXXX...XXX.X....] duration: 6.114814595s FAIL (2025-07-26 23:30:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.114838311s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...............................XXX.X..X.....XXXXX...XXX.X...] duration: 17.322098092s FAIL (2025-07-26 23:30:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.322109243s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.245970893s FAIL (2025-07-26 23:30:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== ) [seg_pos={0 0}, upload=57467]: Begin upload segment... Jul 26 23:29:30.957525843 (59h53m40.680088883s) [seg_pos={0 0}, upload=57467]: Done begin upload segment. Jul 26 23:29:30.957553476 (59h53m40.680116526s) [noise=true , address=45.91.116.8:4045 , piece_id=UHUAGFWKT5D5NEELNHYYRTL6A5UHDMMZI64FJIM5CP4AR4TGF4DA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.957562202 (59h53m40.680125292s) [noise=true , address=104.36.82.16:4294 , piece_id=KR3YHDPWNGHM2Y6YKBZ2N2GZZMBTPAARC77PEXDZPBLUJ3DTEGZA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958059744 (59h53m40.680622784s) [noise=true , address=104.36.82.4:4801 , piece_id=XNGRE732J3DX2WBVP3BVX3PSI3TR62IDCPLOFLUYACO6A4SNMSGQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958425615 (59h53m40.680988665s) [noise=true , address=45.91.116.9:4262 , piece_id=M6T3HG6ESFJOLI3QKNRZNOF4RVVTSM7DHSWSOLXPNQUPIBJTTY4Q, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958455181 (59h53m40.681018221s) [noise=true , address=104.36.82.14:4222 , piece_id=3AEM2C25R2FWVCKTN2UGSY3T2DMVKNV2PSN6Y5AF5IDKXCJA7KBQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958896826 (59h53m40.681459866s) [noise=true , address=104.36.82.13:4186 , piece_id=VUJTVCKV2P5FJAQ7B5ANNVNGZYRRLTSDL4WIL5KTDLA7OR7KOYBQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.958912457 (59h53m40.681475497s) [noise=true , address=104.36.82.16:4293 , piece_id=AZMFGRGMIHM4Z7IQOACCH3PVK47CUA4JKWUBCN435WFGC66YE77Q, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959368789 (59h53m40.681931829s) [noise=true , address=45.91.116.12:4227 , piece_id=SQTXQJKUOTMOMUDCTLTYRETCLJC5QIX25AZSP3H3C4BXYPBSY65A, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959384530 (59h53m40.68194757s) [noise=true , address=77.90.31.14:4662 , piece_id=5OQJBZHAQ42HTKTT7JIT22INFSXZ7EHR4GPYVHMTMQ7EF3HD6XEA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959800195 (59h53m40.682363235s) [noise=true , address=104.36.82.13:4187 , piece_id=LMRHEEM4U6XG4UTJSXHB3QCXQ2FFE7Z6E6LPYTE36NIXVLO26IZA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.959826545 (59h53m40.682389605s) [noise=true , address=104.36.82.2:4729 , piece_id=47JQQJTJQWVAMXIFGMPXV6SDBNMV2EW6CLH4PBTTMLNCOAUNGIOQ, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960240377 (59h53m40.682803427s) [noise=true , address=45.91.116.4:4940 , piece_id=POXONBGOYLCBXHD7RS4DP7F4M4NAIF2YKM5HWRUJSXIOJ3KIK6XA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960268361 (59h53m40.682831411s) [noise=true , address=104.36.82.7:4911 , piece_id=WNIVV36M26PZIJYE3VBTJXRQ3I4YFEJECGA5GLWBZZSI4RB2YJ4A, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960600396 (59h53m40.683163436s) [noise=true , address=104.36.82.11:4114 , piece_id=ASR55YF57A5NT637BY3AEPNAVB2N44AIFCTY6VF54LZGMAIRRKBA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.960659729 (59h53m40.683222769s) [noise=true , address=104.36.82.9:4006 , piece_id=2DSHSBYLKS57RTLGXHBZ6UROCCKXUHYQEAHS7PG4AFCTSXWZE6CA, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:30.961452526 (59h53m40.684015566s) [noise=true , address=104.36.82.2:4732 , piece_id=ZV7U43ZS4JCLCHP27Q276B2BJBXHT4CUVJ6A2NH3GURGUSIJGM4Q, seg_pos={0 0}, upload=57467]: Uploading piece... Jul 26 23:29:32.727099560 (59h53m42.44966259s) [upload=57467]: Next returned nil segment Jul 26 23:29:32.727119649 (59h53m42.449682689s) [upload=57467]: Waiting for error group managing segments... Jul 26 23:29:33.056938693 (59h53m42.779501723s) [noise=true , address=45.91.116.4:4940 , piece_id=POXONBGOYLCBXHD7RS4DP7F4M4NAIF2YKM5HWRUJSXIOJ3KIK6XA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.061739307 (59h53m42.784302347s) [noise=true , address=45.91.116.8:4045 , piece_id=UHUAGFWKT5D5NEELNHYYRTL6A5UHDMMZI64FJIM5CP4AR4TGF4DA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.070898932 (59h53m42.793461962s) [noise=true , address=45.91.116.9:4262 , piece_id=M6T3HG6ESFJOLI3QKNRZNOF4RVVTSM7DHSWSOLXPNQUPIBJTTY4Q, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.071978769 (59h53m42.794541809s) [noise=true , address=45.91.116.12:4227 , piece_id=SQTXQJKUOTMOMUDCTLTYRETCLJC5QIX25AZSP3H3C4BXYPBSY65A, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:33.203596053 (59h53m42.926159093s) [noise=true , address=77.90.31.14:4662 , piece_id=5OQJBZHAQ42HTKTT7JIT22INFSXZ7EHR4GPYVHMTMQ7EF3HD6XEA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: <nil> Jul 26 23:29:59.857576824 (59h54m9.580139864s) [noise=true , address=104.36.82.16:4293 , piece_id=AZMFGRGMIHM4Z7IQOACCH3PVK47CUA4JKWUBCN435WFGC66YE77Q, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.857761588 (59h54m9.580324628s) [noise=true , address=104.36.82.16:4294 , piece_id=KR3YHDPWNGHM2Y6YKBZ2N2GZZMBTPAARC77PEXDZPBLUJ3DTEGZA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.857904701 (59h54m9.580467741s) [noise=true , address=104.36.82.2:4729 , piece_id=47JQQJTJQWVAMXIFGMPXV6SDBNMV2EW6CLH4PBTTMLNCOAUNGIOQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.857991577 (59h54m9.580554687s) [noise=true , address=104.36.82.14:4222 , piece_id=3AEM2C25R2FWVCKTN2UGSY3T2DMVKNV2PSN6Y5AF5IDKXCJA7KBQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858028538 (59h54m9.580591578s) [noise=true , address=104.36.82.13:4186 , piece_id=VUJTVCKV2P5FJAQ7B5ANNVNGZYRRLTSDL4WIL5KTDLA7OR7KOYBQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858061822 (59h54m9.580624862s) [noise=true , address=104.36.82.7:4911 , piece_id=WNIVV36M26PZIJYE3VBTJXRQ3I4YFEJECGA5GLWBZZSI4RB2YJ4A, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858151223 (59h54m9.580714263s) [noise=true , address=104.36.82.13:4187 , piece_id=LMRHEEM4U6XG4UTJSXHB3QCXQ2FFE7Z6E6LPYTE36NIXVLO26IZA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858177193 (59h54m9.580740233s) [noise=true , address=104.36.82.11:4114 , piece_id=ASR55YF57A5NT637BY3AEPNAVB2N44AIFCTY6VF54LZGMAIRRKBA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858210025 (59h54m9.580773065s) [noise=true , address=104.36.82.2:4732 , piece_id=ZV7U43ZS4JCLCHP27Q276B2BJBXHT4CUVJ6A2NH3GURGUSIJGM4Q, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 188.166.225.170:38716->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858221357 (59h54m9.580784397s) [noise=true , address=104.36.82.4:4801 , piece_id=XNGRE732J3DX2WBVP3BVX3PSI3TR62IDCPLOFLUYACO6A4SNMSGQ, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858241405 (59h54m9.580804445s) [noise=true , address=104.36.82.9:4006 , piece_id=2DSHSBYLKS57RTLGXHBZ6UROCCKXUHYQEAHS7PG4AFCTSXWZE6CA, seg_pos={0 0}, upload=57467]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:59.858260843 (59h54m9.580823883s) [seg_pos={0 0}, upload=57467]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:30:00.101767948 (59h54m9.824330978s) [upload=57467]: Done uploading segments... Jul 26 23:30:00.101794078 (59h54m9.824357118s) [upload=57467]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.245994508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [.X.X...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.245926377s FAIL (2025-07-26 23:29:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.245947015s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 52 out of 60 is failed (threshold=3) [XX.X.XX.XX.X.XXXXXXXXX...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.258034307s FAIL (2025-07-26 23:29:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== , piece_id=FCF44UOI6KB6YYOXAJFQXYKHURIHBHV4KU4H63UWKW2R357UO7JA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.201952808 (59h53m2.924515848s) [noise=true , address=79.127.205.233:4633 , piece_id=LIXBVOW3ZT3MMN5JJUATCD4ACWIYHU5CTV46LBCRA4AQYXP5ALNQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202218657 (59h53m2.924781697s) [noise=true , address=45.91.116.14:4194 , piece_id=JZNRF5IGDRMUPLGGUKNSCMD3HXERTD37KOZKKZC26OTUOGEIMHNQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202520945 (59h53m2.925083985s) [noise=true , address=45.91.116.13:4175 , piece_id=37R4ZOXVNUPKA5TWCGUQIY25UUTDRRERXQ2VKHF5XRH5LC4OOZGQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202767176 (59h53m2.925330206s) [noise=true , address=104.36.82.46:4489 , piece_id=VGLV6UBB5TUPIPVVHYE4LUQ7MTVM3RMWARUN4X2N3OUARQ3EIJEQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.202970264 (59h53m2.925533304s) [noise=true , address=79.127.219.38:4006 , piece_id=BQPEWDHZXHPSEKB5IDJMDLSBSEPPTMTZ6UBTQXWLN3G75RVOLLLA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203170648 (59h53m2.925733689s) [noise=true , address=45.91.116.14:4206 , piece_id=IIJ5RBS253DUUXS4HMIE4FFVBAGGEEOBAV3MRK5APZEYPCYLUALQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203371442 (59h53m2.925934482s) [noise=true , address=79.127.205.245:4242 , piece_id=R4N52O3SW5H4SU45G4HTI4GJ36HTNJ5AADX2G4A5EW4SG5O2QR3Q, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203622172 (59h53m2.926185212s) [noise=true , address=104.36.82.47:4546 , piece_id=4L2HVKDNU6YOHWOSW5EPOFTDMSBBSIZBIDRFHFXYHK4NOLB4A6QA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203816554 (59h53m2.926379594s) [noise=true , address=104.36.82.55:4197 , piece_id=RFYPIJTOHPDSSUATKJQND3GAY43FNPKMPOTFKI23BHLIFVMW26IQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.203999103 (59h53m2.926562143s) [noise=true , address=104.36.82.9:4027 , piece_id=QRZNQU47JNKE42REOQTOPEKLTT3RMQJKG42ADW4AK5DKMPCQ7INA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204212271 (59h53m2.926775311s) [noise=true , address=156.146.43.225:4448 , piece_id=67ITBWULB6ZTXA3XSSUBWKUK3P7G6SQLEWAYPGUEOCDMMXLQPIKQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204395862 (59h53m2.926958902s) [noise=true , address=104.36.82.53:4128 , piece_id=ATYIQZITUQIFVWMC7SNA4F56Q2J7PTBZFCI4H42OM4X3HXAZMGPQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204578883 (59h53m2.927141923s) [noise=true , address=45.91.116.14:4222 , piece_id=NRYKVAKFCPFQUNY6B5XIDT73E2NFSKX3DQKRCGNZRO3S44Q3LRFQ, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204781961 (59h53m2.927345001s) [noise=true , address=79.127.220.101:4041 , piece_id=Q5QQT6AI23Y66IN5K45M2R7WMWPWWGWG5CA5J4M6XSCB2BDXVQOA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.204957988 (59h53m2.927521028s) [noise=true , address=104.36.82.54:4162 , piece_id=4VJDYSUKPOOMIAFARIWUNBT4AZZDQ76WA6776H52I3VYBFKV5VCA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:53.209476623 (59h53m2.932039663s) [noise=true , address=77.90.31.16:4764 , piece_id=CDH3FO6MBSSJO67TBUZ45YRW5WUUIATXAAGQLIKTMP3QXRL3RGOA, seg_pos={0 0}, upload=57456]: Uploading piece... Jul 26 23:28:54.297433341 (59h53m4.019996381s) [upload=57456]: Next returned nil segment Jul 26 23:28:54.297452078 (59h53m4.020015118s) [upload=57456]: Waiting for error group managing segments... Jul 26 23:28:54.648486091 (59h53m4.371049121s) [noise=true , address=79.127.205.245:4242 , piece_id=R4N52O3SW5H4SU45G4HTI4GJ36HTNJ5AADX2G4A5EW4SG5O2QR3Q, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:54.748331225 (59h53m4.470894265s) [noise=true , address=79.127.205.233:4633 , piece_id=LIXBVOW3ZT3MMN5JJUATCD4ACWIYHU5CTV46LBCRA4AQYXP5ALNQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:55.064117540 (59h53m4.78668058s) [noise=true , address=79.127.219.38:4006 , piece_id=BQPEWDHZXHPSEKB5IDJMDLSBSEPPTMTZ6UBTQXWLN3G75RVOLLLA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:55.207205590 (59h53m4.929768641s) [noise=true , address=77.90.31.17:4787 , piece_id=FCF44UOI6KB6YYOXAJFQXYKHURIHBHV4KU4H63UWKW2R357UO7JA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:55.743367591 (59h53m5.465930621s) [noise=true , address=45.91.116.14:4222 , piece_id=NRYKVAKFCPFQUNY6B5XIDT73E2NFSKX3DQKRCGNZRO3S44Q3LRFQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 23:28:56.887766172 (59h53m6.610329212s) [noise=true , address=156.146.43.225:4448 , piece_id=67ITBWULB6ZTXA3XSSUBWKUK3P7G6SQLEWAYPGUEOCDMMXLQPIKQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:57.151823643 (59h53m6.874386683s) [noise=true , address=45.91.116.14:4194 , piece_id=JZNRF5IGDRMUPLGGUKNSCMD3HXERTD37KOZKKZC26OTUOGEIMHNQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:57.929273047 (59h53m7.651836127s) [noise=true , address=45.91.116.13:4175 , piece_id=37R4ZOXVNUPKA5TWCGUQIY25UUTDRRERXQ2VKHF5XRH5LC4OOZGQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:28:58.070012284 (59h53m7.792575314s) [noise=true , address=45.91.116.14:4206 , piece_id=IIJ5RBS253DUUXS4HMIE4FFVBAGGEEOBAV3MRK5APZEYPCYLUALQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 26 23:28:58.184864451 (59h53m7.907427491s) [noise=true , address=79.127.220.101:4041 , piece_id=Q5QQT6AI23Y66IN5K45M2R7WMWPWWGWG5CA5J4M6XSCB2BDXVQOA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:29:03.615820933 (59h53m13.338384003s) [noise=true , address=77.90.31.13:4634 , piece_id=C6ECRJ6O63TCTDOAB4ICSGTY3LR2Q433MNJFRYEWWNG3255K6CUQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: <nil> Jul 26 23:29:52.120011425 (59h54m1.842574465s) [noise=true , address=104.36.82.46:4489 , piece_id=VGLV6UBB5TUPIPVVHYE4LUQ7MTVM3RMWARUN4X2N3OUARQ3EIJEQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: write tcp 188.166.225.170:54066->104.36.82.46:4489: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120100366 (59h54m1.842663406s) [noise=true , address=104.36.82.47:4546 , piece_id=4L2HVKDNU6YOHWOSW5EPOFTDMSBBSIZBIDRFHFXYHK4NOLB4A6QA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: write tcp 188.166.225.170:60352->104.36.82.47:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120153206 (59h54m1.842716246s) [noise=true , address=104.36.82.55:4197 , piece_id=RFYPIJTOHPDSSUATKJQND3GAY43FNPKMPOTFKI23BHLIFVMW26IQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 188.166.225.170:33082->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120250443 (59h54m1.842813483s) [noise=true , address=77.90.31.16:4764 , piece_id=CDH3FO6MBSSJO67TBUZ45YRW5WUUIATXAAGQLIKTMP3QXRL3RGOA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: write tcp 188.166.225.170:53384->77.90.31.16:4764: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120321940 (59h54m1.84288497s) [noise=true , address=104.36.82.9:4027 , piece_id=QRZNQU47JNKE42REOQTOPEKLTT3RMQJKG42ADW4AK5DKMPCQ7INA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:1SWk13XvkrCYwKcdZ46SK2THQvhsD6pf4hWJKgARztB9qwfMZU, address:104.36.82.9:4027): protocol: write tcp 188.166.225.170:43060->104.36.82.9:4027: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120342729 (59h54m1.842905769s) [noise=true , address=104.36.82.53:4128 , piece_id=ATYIQZITUQIFVWMC7SNA4F56Q2J7PTBZFCI4H42OM4X3HXAZMGPQ, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 188.166.225.170:47254->104.36.82.53:4128: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120396602 (59h54m1.842959642s) [noise=true , address=104.36.82.54:4162 , piece_id=4VJDYSUKPOOMIAFARIWUNBT4AZZDQ76WA6776H52I3VYBFKV5VCA, seg_pos={0 0}, upload=57456]: Done uploading piece. err: ecclient: upload failed (node:12hV2HLvT4YUVqeM2eAVe8p8UFVsZK7MYN95YinC8KMss9sKDXt, address:104.36.82.54:4162): protocol: rpc: dial tcp 104.36.82.54:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:52.120416951 (59h54m1.842979991s) [seg_pos={0 0}, upload=57456]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:29:52.375798713 (59h54m2.098361753s) [upload=57456]: Done uploading segments... Jul 26 23:29:52.375820325 (59h54m2.098383365s) [upload=57456]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.258050348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.875904769s FAIL (2025-07-26 23:29:52 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.875930078s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.828177846s FAIL (2025-07-26 23:29:28 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 1.828202903s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.........X...........X.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.229738813s FAIL (2025-07-26 23:29:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== RJ2SZPA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385005830 (59h53m6.10756886s) [noise=true , address=77.90.31.13:4626 , piece_id=YPPZQ45IRVUZR75V3NL4BPEHJBRDW7EVGTZTFDWUSOCFZC6ZPRJQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385329399 (59h53m6.107892439s) [noise=true , address=104.36.82.4:4801 , piece_id=LDYPYGMJM6PUP6ROZVF5BEDMFZEVBZZUCP6X77GHFX2JK2ALJ3HA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385336202 (59h53m6.107899232s) [noise=true , address=104.36.82.7:4909 , piece_id=4DVNFWI5FXUUWHBGKRPP6KPW4AEG3QWIYDI75KI4KUOVSX2KI26Q, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.385993239 (59h53m6.108556279s) [noise=true , address=104.36.82.5:4839 , piece_id=NF3EMKQNTW3MZEQ4PI3GBSUSB7JV3737IGB24FLQYSHHERPUQJNA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.386355362 (59h53m6.108918402s) [noise=true , address=104.36.82.12:4150 , piece_id=LQPJI7TBNZKNQAUOV3JE6PXC4KCOD6WOWYR3GC7CSNAZL2J64O3A, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.386599499 (59h53m6.109162539s) [noise=true , address=104.36.82.13:4185 , piece_id=2XWX753JZZWFBFGPYHKXPSM3GVKRSEO2M5CHS75Z4J3K35QCCOSQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387006628 (59h53m6.109569668s) [noise=true , address=45.91.116.6:4974 , piece_id=5UCBKIHHG56VB4HRGVKZZSNHK2UWATQ3JD5L3I5VLG5YJXGSQDSQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387276013 (59h53m6.109839073s) [noise=true , address=104.36.82.10:4079 , piece_id=XDDXUCTOM7UDYO7SX5VEHFSML6O2WJPAO5SGNELCIRB7RVIAH47A, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387658394 (59h53m6.110221434s) [noise=true , address=104.36.82.16:4293 , piece_id=KDBNFAZRNA53O2SHLEMCZ7VFR5T32WJIN3WUUPWM2FI4Z2QHJRQA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.387900097 (59h53m6.110463137s) [noise=true , address=77.90.31.10:4518 , piece_id=GPND4PW3X6STFXHVT24XHTP3JMVBDZP6JZGGXW4B6Z4LPZ3K3XFQ, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.388186605 (59h53m6.110749655s) [noise=true , address=104.36.82.9:4005 , piece_id=UZHDIVV6LSRQLYZ7S7643D4CJZ36WVTWBYRPFLWTTWXCE6A2WZ3Q, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.388387620 (59h53m6.11095066s) [noise=true , address=104.36.82.11:4113 , piece_id=YZ2TYDLKHX25JU5ZYOMHPNAEBXIKRAS4KR45GTFZS3S32YBFUPLA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:56.388601920 (59h53m6.11116496s) [noise=true , address=77.90.31.16:4733 , piece_id=X6JP7VU4IIZKCMOKCNUKB65UTPUXVTQU7JLPB2OMUFL7XYO6UVNA, seg_pos={0 0}, upload=57458]: Uploading piece... Jul 26 23:28:58.202080039 (59h53m7.924643079s) [upload=57458]: Next returned nil segment Jul 26 23:28:58.202096922 (59h53m7.924659962s) [upload=57458]: Waiting for error group managing segments... Jul 26 23:28:58.509145339 (59h53m8.231708369s) [noise=true , address=45.91.116.6:4974 , piece_id=5UCBKIHHG56VB4HRGVKZZSNHK2UWATQ3JD5L3I5VLG5YJXGSQDSQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: <nil> Jul 26 23:28:58.587316152 (59h53m8.309879192s) [noise=true , address=77.90.31.10:4518 , piece_id=GPND4PW3X6STFXHVT24XHTP3JMVBDZP6JZGGXW4B6Z4LPZ3K3XFQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: <nil> Jul 26 23:29:00.003774366 (59h53m9.726337416s) [noise=true , address=77.90.31.13:4626 , piece_id=YPPZQ45IRVUZR75V3NL4BPEHJBRDW7EVGTZTFDWUSOCFZC6ZPRJQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: <nil> Jul 26 23:29:25.514764348 (59h53m35.237327378s) [noise=true , address=104.36.82.9:4005 , piece_id=UZHDIVV6LSRQLYZ7S7643D4CJZ36WVTWBYRPFLWTTWXCE6A2WZ3Q, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: write tcp 188.166.225.170:45048->104.36.82.9:4005: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514822198 (59h53m35.237385238s) [noise=true , address=104.36.82.3:4765 , piece_id=BEOG7KHUHS74JKMFI6EUXM6UKGB5M2YII4AGTDURMHRTPVEHNSZA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514862476 (59h53m35.237425516s) [noise=true , address=104.36.82.2:4729 , piece_id=ZNO2MUPZHYH3DGX7XM35IJITSWY6DBT3HVCN3OZ3IOU4BRJ2SZPA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:34642->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514894948 (59h53m35.237457988s) [noise=true , address=104.36.82.4:4801 , piece_id=LDYPYGMJM6PUP6ROZVF5BEDMFZEVBZZUCP6X77GHFX2JK2ALJ3HA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514928131 (59h53m35.237491171s) [noise=true , address=104.36.82.10:4077 , piece_id=CXNP6AZHBEAJBZHSXW6BVAVUJMDULEFLVLG3W6GI2SJLDRIU5PEA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 188.166.225.170:41600->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.514962617 (59h53m35.237525657s) [noise=true , address=104.36.82.7:4909 , piece_id=4DVNFWI5FXUUWHBGKRPP6KPW4AEG3QWIYDI75KI4KUOVSX2KI26Q, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:36100->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515017141 (59h53m35.237580181s) [noise=true , address=104.36.82.5:4839 , piece_id=NF3EMKQNTW3MZEQ4PI3GBSUSB7JV3737IGB24FLQYSHHERPUQJNA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 188.166.225.170:53076->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515051326 (59h53m35.237614366s) [noise=true , address=104.36.82.12:4150 , piece_id=LQPJI7TBNZKNQAUOV3JE6PXC4KCOD6WOWYR3GC7CSNAZL2J64O3A, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515098807 (59h53m35.237661837s) [noise=true , address=104.36.82.16:4293 , piece_id=KDBNFAZRNA53O2SHLEMCZ7VFR5T32WJIN3WUUPWM2FI4Z2QHJRQA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:44688->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515129135 (59h53m35.237692205s) [noise=true , address=104.36.82.11:4113 , piece_id=YZ2TYDLKHX25JU5ZYOMHPNAEBXIKRAS4KR45GTFZS3S32YBFUPLA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:52024->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515170936 (59h53m35.237733976s) [noise=true , address=77.90.31.16:4733 , piece_id=X6JP7VU4IIZKCMOKCNUKB65UTPUXVTQU7JLPB2OMUFL7XYO6UVNA, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: rpc: dial tcp 77.90.31.16:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515203047 (59h53m35.237766077s) [noise=true , address=104.36.82.13:4185 , piece_id=2XWX753JZZWFBFGPYHKXPSM3GVKRSEO2M5CHS75Z4J3K35QCCOSQ, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515304642 (59h53m35.237867722s) [noise=true , address=104.36.82.10:4079 , piece_id=XDDXUCTOM7UDYO7SX5VEHFSML6O2WJPAO5SGNELCIRB7RVIAH47A, seg_pos={0 0}, upload=57458]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: write tcp 188.166.225.170:48046->104.36.82.10:4079: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:25.515338236 (59h53m35.237901276s) [seg_pos={0 0}, upload=57458]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Jul 26 23:29:25.741941816 (59h53m35.464504846s) [upload=57458]: Done uploading segments... Jul 26 23:29:25.741968177 (59h53m35.464531217s) [upload=57458]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 30.229760525s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)