Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X.XX..X] duration: 30.002693186s FAIL (2025-09-06 15:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:49:02.503680029 (2h20m38.543253149s) [upload=2048]: Starting upload Sep 6 15:49:02.503734543 (2h20m38.543307663s) [upload=2048]: Done starting upload Sep 6 15:49:02.503822162 (2h20m38.543395282s) [upload=2048]: Uploading segments... Sep 6 15:49:02.503894580 (2h20m38.5434677s) [upload=2048]: Got next segment. Inline: false Sep 6 15:49:02.503920339 (2h20m38.543493459s) [upload=2048]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:49:02.503925278 (2h20m38.543498398s) [upload=2048]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:49:02.747920277 (2h20m38.787493397s) [seg_pos={0 0}, upload=2048]: Begin upload segment... Sep 6 15:49:02.747962387 (2h20m38.787535507s) [seg_pos={0 0}, upload=2048]: Done begin upload segment. Sep 6 15:49:02.747983418 (2h20m38.787556528s) [noise=true , address=45.91.116.12:4225 , piece_id=5MVIUDSZCEHXH5QGOVTFHIP4LYCYF3Y2LFTR442A6I6LKASIC53A, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.748376439 (2h20m38.787949559s) [noise=true , address=45.91.116.12:4226 , piece_id=POUDFASAC3XDDZ4ETLGJYFI3FCRA5Z3YEWNJ7OYYQN3ZQI2NK4LQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.748672175 (2h20m38.788245296s) [noise=true , address=45.91.116.5:4299 , piece_id=2FWCEXCH4E6KE5BAJDQBGB2JAPK7KQDD74H2HKTUXSECFE42VQBQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.748952061 (2h20m38.788525181s) [noise=true , address=45.91.116.5:4298 , piece_id=TTE5NFTGMQGEP3IUREVT6TNJWEY3WBK5RKAXU7VRJ6XNPGLHRB4Q, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.749292873 (2h20m38.788865994s) [noise=true , address=45.91.116.7:4010 , piece_id=4XLDLDPVFYWPKXYQ4YRXH53OJUBOFFGKXZ5NCRY5VERY33BHRAQA, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.749583720 (2h20m38.78915684s) [noise=true , address=45.91.116.10:4082 , piece_id=7I7GC35NJ75YXLF7MRXG2KTWKTCIEJE6ZLM3GQG7WQVZLDEJ7YAA, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.751712252 (2h20m38.791285373s) [noise=true , address=45.91.116.11:4117 , piece_id=BF6VHYKOQAF6Z3E5YFJNOKZZ5KSU7XVKVFWRTJP56JJ553ESGW6Q, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.751971078 (2h20m38.791544198s) [noise=true , address=77.90.31.10:4517 , piece_id=VTVDDHSHFCSDE3OEPKRMYBUKI3JJU657KF6HHO4H5HFFMXILVXXQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.752196910 (2h20m38.79177003s) [noise=true , address=77.90.31.14:4663 , piece_id=LMMT3TETBH2C6LHYLPDXUOQOU6EUERXFFBU6YGTZOJVJRJS362KQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.752451547 (2h20m38.792024667s) [noise=true , address=45.91.116.6:4975 , piece_id=S5B2UEFY243G3IRVKLHA3HFCJOW4HVO6O2TBHYTOW5NCTY72HPUQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.752693039 (2h20m38.792266159s) [noise=true , address=45.91.116.7:4011 , piece_id=TKWYVBPAEIGT3CPCGWCBWSNARU2WP4E63JXKIZORVQE4EURDFEIA, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.752965360 (2h20m38.792538481s) [noise=true , address=45.91.116.4:4939 , piece_id=S5BUOOYX2GB6JRHKWYNLEZUSQFEUFGDYNK2WPJUCV33QT6EPNMTQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.753214307 (2h20m38.792787427s) [noise=true , address=77.90.31.19:4843 , piece_id=XN7LXNUO2WXWPKB2GGZKROMQ455PKMPZYEXN7HYJ4WZBHG3FP44Q, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.753511375 (2h20m38.793084495s) [noise=true , address=77.90.31.16:4735 , piece_id=G5ZNYGXQELSGVVIIZJHU7SPVRFYD2AW75QXOLYIPATCB6GLTH74A, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.753736416 (2h20m38.793309536s) [noise=true , address=45.91.116.10:4083 , piece_id=F6YMH34PZHPPRZHSJGGA5DILDOS2C6DQL5WE7I4C7YVCIJFI6NBQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:02.754017144 (2h20m38.793590264s) [noise=true , address=45.91.116.6:4973 , piece_id=7VWR5C2SP2TYYLBYCAX27A65U5YLQVBH4PJKYYFEUL75ZTRMZVDQ, seg_pos={0 0}, upload=2048]: Uploading piece... Sep 6 15:49:04.516560153 (2h20m40.556133263s) [upload=2048]: Next returned nil segment Sep 6 15:49:04.516579680 (2h20m40.5561528s) [upload=2048]: Waiting for error group managing segments... Sep 6 15:49:04.878525999 (2h20m40.918099119s) [noise=true , address=45.91.116.4:4939 , piece_id=S5BUOOYX2GB6JRHKWYNLEZUSQFEUFGDYNK2WPJUCV33QT6EPNMTQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:04.897152331 (2h20m40.936725451s) [noise=true , address=45.91.116.10:4083 , piece_id=F6YMH34PZHPPRZHSJGGA5DILDOS2C6DQL5WE7I4C7YVCIJFI6NBQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:04.960396018 (2h20m40.999969148s) [noise=true , address=45.91.116.6:4973 , piece_id=7VWR5C2SP2TYYLBYCAX27A65U5YLQVBH4PJKYYFEUL75ZTRMZVDQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:04.973530141 (2h20m41.013103271s) [noise=true , address=45.91.116.5:4299 , piece_id=2FWCEXCH4E6KE5BAJDQBGB2JAPK7KQDD74H2HKTUXSECFE42VQBQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:04.987635521 (2h20m41.027208641s) [noise=true , address=45.91.116.11:4117 , piece_id=BF6VHYKOQAF6Z3E5YFJNOKZZ5KSU7XVKVFWRTJP56JJ553ESGW6Q, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:04.988525805 (2h20m41.028098925s) [noise=true , address=45.91.116.7:4010 , piece_id=4XLDLDPVFYWPKXYQ4YRXH53OJUBOFFGKXZ5NCRY5VERY33BHRAQA, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.001758315 (2h20m41.041331456s) [noise=true , address=77.90.31.16:4735 , piece_id=G5ZNYGXQELSGVVIIZJHU7SPVRFYD2AW75QXOLYIPATCB6GLTH74A, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.029469111 (2h20m41.069042241s) [noise=true , address=45.91.116.12:4226 , piece_id=POUDFASAC3XDDZ4ETLGJYFI3FCRA5Z3YEWNJ7OYYQN3ZQI2NK4LQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.037230923 (2h20m41.076804063s) [noise=true , address=45.91.116.6:4975 , piece_id=S5B2UEFY243G3IRVKLHA3HFCJOW4HVO6O2TBHYTOW5NCTY72HPUQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.044387918 (2h20m41.083961038s) [noise=true , address=77.90.31.19:4843 , piece_id=XN7LXNUO2WXWPKB2GGZKROMQ455PKMPZYEXN7HYJ4WZBHG3FP44Q, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.107201653 (2h20m41.146774773s) [noise=true , address=45.91.116.7:4011 , piece_id=TKWYVBPAEIGT3CPCGWCBWSNARU2WP4E63JXKIZORVQE4EURDFEIA, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.619359695 (2h20m41.658932815s) [noise=true , address=45.91.116.10:4082 , piece_id=7I7GC35NJ75YXLF7MRXG2KTWKTCIEJE6ZLM3GQG7WQVZLDEJ7YAA, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.665351600 (2h20m41.70492472s) [noise=true , address=77.90.31.14:4663 , piece_id=LMMT3TETBH2C6LHYLPDXUOQOU6EUERXFFBU6YGTZOJVJRJS362KQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: <nil> Sep 6 15:49:05.665381167 (2h20m41.704954277s) [seg_pos={0 0}, upload=2048]: Segment reached optimal threshold of 13 pieces. Sep 6 15:49:05.665516776 (2h20m41.705089896s) [noise=true , address=77.90.31.10:4517 , piece_id=VTVDDHSHFCSDE3OEPKRMYBUKI3JJU657KF6HHO4H5HFFMXILVXXQ, seg_pos={0 0}, upload=2048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 6 15:49:05.665626727 (2h20m41.705199847s) [noise=true , address=45.91.116.5:4298 , piece_id=TTE5NFTGMQGEP3IUREVT6TNJWEY3WBK5RKAXU7VRJ6XNPGLHRB4Q, seg_pos={0 0}, upload=2048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Sep 6 15:49:05.665694747 (2h20m41.705267867s) [noise=true , address=45.91.116.12:4225 , piece_id=5MVIUDSZCEHXH5QGOVTFHIP4LYCYF3Y2LFTR442A6I6LKASIC53A, seg_pos={0 0}, upload=2048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:35652->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:35652->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35652->45.91.116.12:4225: use of closed network connection Sep 6 15:49:05.665712080 (2h20m41.7052852s) [seg_pos={0 0}, upload=2048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:49:05.665776083 (2h20m41.705349203s) [upload=2048]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:49:05.665779840 (2h20m41.70535296s) [upload=2048]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:49:05.966634590 (2h20m42.00620771s) [upload=2048]: Done uploading segments... Sep 6 15:49:05.966652524 (2h20m42.006225644s) [upload=2048]: Upload finished. err: <nil> Duration: 30.002723353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.988158121s FAIL (2025-09-06 15:49:30 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.988183038s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..............X...] duration: 6.800272511s FAIL (2025-09-06 15:49:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.800293411s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..............X..] duration: 6.739481138s FAIL (2025-09-06 15:48:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.73950376s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.175265103s FAIL (2025-09-06 15:48:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.175293969s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.502408625s FAIL (2025-09-06 15:48:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.502431758s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X.XX..] duration: 8.959338601s FAIL (2025-09-06 15:48:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:47:01.332418454 (2h18m37.371991574s) [upload=2013]: Starting upload Sep 6 15:47:01.332472658 (2h18m37.372045778s) [upload=2013]: Done starting upload Sep 6 15:47:01.332545006 (2h18m37.372118126s) [upload=2013]: Uploading segments... Sep 6 15:47:01.332656139 (2h18m37.372229259s) [upload=2013]: Got next segment. Inline: false Sep 6 15:47:01.332700954 (2h18m37.372274074s) [upload=2013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:47:01.332706404 (2h18m37.372279524s) [upload=2013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:47:03.864074672 (2h18m39.903647822s) [seg_pos={0 0}, upload=2013]: Begin upload segment... Sep 6 15:47:03.864137852 (2h18m39.903710972s) [seg_pos={0 0}, upload=2013]: Done begin upload segment. Sep 6 15:47:03.864168751 (2h18m39.903741871s) [noise=true , address=45.91.116.5:4298 , piece_id=TB4AUZC27GKH4A7FXMKRUNJY2LY7B5VCHHQVDSC64KOCVSTQJBNQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.864205001 (2h18m39.903778111s) [noise=true , address=45.91.116.13:4153 , piece_id=KFELQSQS7IFE7LTESU4MMF3EC4AKA2EPIKSYTLDLHA4YQ3WHK6CA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.864705579 (2h18m39.904278709s) [noise=true , address=77.90.31.12:4592 , piece_id=6DNDMYDDDXFNDHFKSXMXY6GK2FL7ZPM44HFUK6UWAI5VYLF3MIJA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.865171360 (2h18m39.9047445s) [noise=true , address=45.91.116.13:4155 , piece_id=CKCFX75SIP63OM5WM2LUGMTS3PUEEJLZAA3F437T7SIUKGAVEPPQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.865529185 (2h18m39.905102305s) [noise=true , address=77.90.31.17:4770 , piece_id=V7NOD2DUP2Y4AMCBJMYJWZG2MRIO3WXS7GH6NX47FAHRLP7SPL6A, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866349194 (2h18m39.905922314s) [noise=true , address=77.90.31.18:4807 , piece_id=N7A3YJCCGHUG43CEPFFMAJNAWT34M3F7O3BOYEULM7GUKCZYMYHA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866634881 (2h18m39.906208001s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZTDSLWTJCO7RZJ2J3YI67I6V37PXTA5QADYP6WGRYWFT7DSTR7Q, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866889548 (2h18m39.906462668s) [noise=true , address=45.91.116.11:4119 , piece_id=KJSC7TTU4PYDY464PLGOJKY6EKXWVMEU2B4TTA5GQTEMB453RMTQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867192718 (2h18m39.906765838s) [noise=true , address=77.90.31.14:4663 , piece_id=7WMBOWSGQJEI2NMRPLVZLU4NPQRLXS4RDM5LUF2D3F6DE373CDCA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867465400 (2h18m39.90703852s) [noise=true , address=77.90.31.18:4805 , piece_id=F3ULLBFHPRJICXJYXFA3NWZ42QIXTYNUWP37CDKF7OTOSQVZHKTA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867658780 (2h18m39.9072319s) [noise=true , address=77.90.31.15:4697 , piece_id=BV5JEJQFK36WR5SWQKPH2B5SUWHFQFUWHDG3ZNAWFHX6J63G7WAA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868116847 (2h18m39.907689977s) [noise=true , address=77.90.31.13:4628 , piece_id=QOZLWJHDST3QWVGIQJNWUWFHPENRJBOWDHCOIXK25TIIVTDBIVFQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868459953 (2h18m39.908033073s) [noise=true , address=77.90.31.17:4771 , piece_id=7DHNZZIUU5WY4YGK5SXZMFGENELXMY6UW62HINBQMMHHAJKWVBMQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868809081 (2h18m39.908382201s) [noise=true , address=77.90.31.13:4627 , piece_id=L5ICEF3FDFE2E5635AEXOKY43CRLMZIDLIVG7AY5GSO2M67NYZCQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.869168819 (2h18m39.908741939s) [noise=true , address=77.90.31.21:4879 , piece_id=PARXJAP4ZTYRCBNGSEMEFFQ4HL2H5DREFY4FTK77O3OUHJA4BJAA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.869481828 (2h18m39.909054949s) [noise=true , address=77.90.31.10:4518 , piece_id=RRCMSPFPOBAJERD7M4KK6RW5ODXZ2FQCZIOQSJ3M6WJ6CRTPUXZQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:07.678956418 (2h18m43.718529538s) [upload=2013]: Next returned nil segment Sep 6 15:47:07.678972889 (2h18m43.718546009s) [upload=2013]: Waiting for error group managing segments... Sep 6 15:47:09.112289519 (2h18m45.151862639s) [noise=true , address=45.91.116.13:4153 , piece_id=KFELQSQS7IFE7LTESU4MMF3EC4AKA2EPIKSYTLDLHA4YQ3WHK6CA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.354792813 (2h18m45.394365933s) [noise=true , address=77.90.31.18:4805 , piece_id=F3ULLBFHPRJICXJYXFA3NWZ42QIXTYNUWP37CDKF7OTOSQVZHKTA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.406791473 (2h18m45.446364593s) [noise=true , address=77.90.31.12:4592 , piece_id=6DNDMYDDDXFNDHFKSXMXY6GK2FL7ZPM44HFUK6UWAI5VYLF3MIJA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.558598698 (2h18m45.598171818s) [noise=true , address=77.90.31.14:4663 , piece_id=7WMBOWSGQJEI2NMRPLVZLU4NPQRLXS4RDM5LUF2D3F6DE373CDCA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.578082551 (2h18m45.617655671s) [noise=true , address=77.90.31.13:4628 , piece_id=QOZLWJHDST3QWVGIQJNWUWFHPENRJBOWDHCOIXK25TIIVTDBIVFQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.800647251 (2h18m45.840220371s) [noise=true , address=77.90.31.13:4627 , piece_id=L5ICEF3FDFE2E5635AEXOKY43CRLMZIDLIVG7AY5GSO2M67NYZCQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.922688218 (2h18m45.962261338s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZTDSLWTJCO7RZJ2J3YI67I6V37PXTA5QADYP6WGRYWFT7DSTR7Q, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.930649503 (2h18m45.970222613s) [noise=true , address=77.90.31.21:4879 , piece_id=PARXJAP4ZTYRCBNGSEMEFFQ4HL2H5DREFY4FTK77O3OUHJA4BJAA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.728726764 (2h18m46.768299884s) [noise=true , address=45.91.116.11:4119 , piece_id=KJSC7TTU4PYDY464PLGOJKY6EKXWVMEU2B4TTA5GQTEMB453RMTQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.923455901 (2h18m46.963029021s) [noise=true , address=45.91.116.5:4298 , piece_id=TB4AUZC27GKH4A7FXMKRUNJY2LY7B5VCHHQVDSC64KOCVSTQJBNQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.930888403 (2h18m46.970461523s) [noise=true , address=77.90.31.15:4697 , piece_id=BV5JEJQFK36WR5SWQKPH2B5SUWHFQFUWHDG3ZNAWFHX6J63G7WAA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:11.061240724 (2h18m47.100813844s) [noise=true , address=77.90.31.18:4807 , piece_id=N7A3YJCCGHUG43CEPFFMAJNAWT34M3F7O3BOYEULM7GUKCZYMYHA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:12.512126805 (2h18m48.551699925s) [noise=true , address=45.91.116.13:4155 , piece_id=CKCFX75SIP63OM5WM2LUGMTS3PUEEJLZAA3F437T7SIUKGAVEPPQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:12.512155750 (2h18m48.55172887s) [seg_pos={0 0}, upload=2013]: Segment reached optimal threshold of 13 pieces. Sep 6 15:47:12.512396922 (2h18m48.551970042s) [noise=true , address=77.90.31.17:4770 , piece_id=V7NOD2DUP2Y4AMCBJMYJWZG2MRIO3WXS7GH6NX47FAHRLP7SPL6A, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45210->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 6 15:47:12.512444063 (2h18m48.552017183s) [noise=true , address=77.90.31.10:4518 , piece_id=RRCMSPFPOBAJERD7M4KK6RW5ODXZ2FQCZIOQSJ3M6WJ6CRTPUXZQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection Sep 6 15:47:12.512476094 (2h18m48.552049214s) [noise=true , address=77.90.31.17:4771 , piece_id=7DHNZZIUU5WY4YGK5SXZMFGENELXMY6UW62HINBQMMHHAJKWVBMQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:41736->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 6 15:47:12.512502023 (2h18m48.552075133s) [seg_pos={0 0}, upload=2013]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:47:12.512574802 (2h18m48.552147922s) [upload=2013]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:47:12.512593177 (2h18m48.552166307s) [upload=2013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:47:13.454815412 (2h18m49.494388533s) [upload=2013]: Done uploading segments... Sep 6 15:47:13.454834750 (2h18m49.49440787s) [upload=2013]: Upload finished. err: <nil> Duration: 8.959364521s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..............X.] duration: 6.789651158s FAIL (2025-09-06 15:47:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.789671887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X.XX.] duration: 14.562759673s FAIL (2025-09-06 15:47:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:47:01.332418454 (2h18m37.371991574s) [upload=2013]: Starting upload Sep 6 15:47:01.332472658 (2h18m37.372045778s) [upload=2013]: Done starting upload Sep 6 15:47:01.332545006 (2h18m37.372118126s) [upload=2013]: Uploading segments... Sep 6 15:47:01.332656139 (2h18m37.372229259s) [upload=2013]: Got next segment. Inline: false Sep 6 15:47:01.332700954 (2h18m37.372274074s) [upload=2013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:47:01.332706404 (2h18m37.372279524s) [upload=2013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:47:03.864074672 (2h18m39.903647822s) [seg_pos={0 0}, upload=2013]: Begin upload segment... Sep 6 15:47:03.864137852 (2h18m39.903710972s) [seg_pos={0 0}, upload=2013]: Done begin upload segment. Sep 6 15:47:03.864168751 (2h18m39.903741871s) [noise=true , address=45.91.116.5:4298 , piece_id=TB4AUZC27GKH4A7FXMKRUNJY2LY7B5VCHHQVDSC64KOCVSTQJBNQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.864205001 (2h18m39.903778111s) [noise=true , address=45.91.116.13:4153 , piece_id=KFELQSQS7IFE7LTESU4MMF3EC4AKA2EPIKSYTLDLHA4YQ3WHK6CA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.864705579 (2h18m39.904278709s) [noise=true , address=77.90.31.12:4592 , piece_id=6DNDMYDDDXFNDHFKSXMXY6GK2FL7ZPM44HFUK6UWAI5VYLF3MIJA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.865171360 (2h18m39.9047445s) [noise=true , address=45.91.116.13:4155 , piece_id=CKCFX75SIP63OM5WM2LUGMTS3PUEEJLZAA3F437T7SIUKGAVEPPQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.865529185 (2h18m39.905102305s) [noise=true , address=77.90.31.17:4770 , piece_id=V7NOD2DUP2Y4AMCBJMYJWZG2MRIO3WXS7GH6NX47FAHRLP7SPL6A, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866349194 (2h18m39.905922314s) [noise=true , address=77.90.31.18:4807 , piece_id=N7A3YJCCGHUG43CEPFFMAJNAWT34M3F7O3BOYEULM7GUKCZYMYHA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866634881 (2h18m39.906208001s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZTDSLWTJCO7RZJ2J3YI67I6V37PXTA5QADYP6WGRYWFT7DSTR7Q, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866889548 (2h18m39.906462668s) [noise=true , address=45.91.116.11:4119 , piece_id=KJSC7TTU4PYDY464PLGOJKY6EKXWVMEU2B4TTA5GQTEMB453RMTQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867192718 (2h18m39.906765838s) [noise=true , address=77.90.31.14:4663 , piece_id=7WMBOWSGQJEI2NMRPLVZLU4NPQRLXS4RDM5LUF2D3F6DE373CDCA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867465400 (2h18m39.90703852s) [noise=true , address=77.90.31.18:4805 , piece_id=F3ULLBFHPRJICXJYXFA3NWZ42QIXTYNUWP37CDKF7OTOSQVZHKTA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867658780 (2h18m39.9072319s) [noise=true , address=77.90.31.15:4697 , piece_id=BV5JEJQFK36WR5SWQKPH2B5SUWHFQFUWHDG3ZNAWFHX6J63G7WAA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868116847 (2h18m39.907689977s) [noise=true , address=77.90.31.13:4628 , piece_id=QOZLWJHDST3QWVGIQJNWUWFHPENRJBOWDHCOIXK25TIIVTDBIVFQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868459953 (2h18m39.908033073s) [noise=true , address=77.90.31.17:4771 , piece_id=7DHNZZIUU5WY4YGK5SXZMFGENELXMY6UW62HINBQMMHHAJKWVBMQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868809081 (2h18m39.908382201s) [noise=true , address=77.90.31.13:4627 , piece_id=L5ICEF3FDFE2E5635AEXOKY43CRLMZIDLIVG7AY5GSO2M67NYZCQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.869168819 (2h18m39.908741939s) [noise=true , address=77.90.31.21:4879 , piece_id=PARXJAP4ZTYRCBNGSEMEFFQ4HL2H5DREFY4FTK77O3OUHJA4BJAA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.869481828 (2h18m39.909054949s) [noise=true , address=77.90.31.10:4518 , piece_id=RRCMSPFPOBAJERD7M4KK6RW5ODXZ2FQCZIOQSJ3M6WJ6CRTPUXZQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:07.678956418 (2h18m43.718529538s) [upload=2013]: Next returned nil segment Sep 6 15:47:07.678972889 (2h18m43.718546009s) [upload=2013]: Waiting for error group managing segments... Sep 6 15:47:09.112289519 (2h18m45.151862639s) [noise=true , address=45.91.116.13:4153 , piece_id=KFELQSQS7IFE7LTESU4MMF3EC4AKA2EPIKSYTLDLHA4YQ3WHK6CA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.354792813 (2h18m45.394365933s) [noise=true , address=77.90.31.18:4805 , piece_id=F3ULLBFHPRJICXJYXFA3NWZ42QIXTYNUWP37CDKF7OTOSQVZHKTA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.406791473 (2h18m45.446364593s) [noise=true , address=77.90.31.12:4592 , piece_id=6DNDMYDDDXFNDHFKSXMXY6GK2FL7ZPM44HFUK6UWAI5VYLF3MIJA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.558598698 (2h18m45.598171818s) [noise=true , address=77.90.31.14:4663 , piece_id=7WMBOWSGQJEI2NMRPLVZLU4NPQRLXS4RDM5LUF2D3F6DE373CDCA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.578082551 (2h18m45.617655671s) [noise=true , address=77.90.31.13:4628 , piece_id=QOZLWJHDST3QWVGIQJNWUWFHPENRJBOWDHCOIXK25TIIVTDBIVFQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.800647251 (2h18m45.840220371s) [noise=true , address=77.90.31.13:4627 , piece_id=L5ICEF3FDFE2E5635AEXOKY43CRLMZIDLIVG7AY5GSO2M67NYZCQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.922688218 (2h18m45.962261338s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZTDSLWTJCO7RZJ2J3YI67I6V37PXTA5QADYP6WGRYWFT7DSTR7Q, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.930649503 (2h18m45.970222613s) [noise=true , address=77.90.31.21:4879 , piece_id=PARXJAP4ZTYRCBNGSEMEFFQ4HL2H5DREFY4FTK77O3OUHJA4BJAA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.728726764 (2h18m46.768299884s) [noise=true , address=45.91.116.11:4119 , piece_id=KJSC7TTU4PYDY464PLGOJKY6EKXWVMEU2B4TTA5GQTEMB453RMTQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.923455901 (2h18m46.963029021s) [noise=true , address=45.91.116.5:4298 , piece_id=TB4AUZC27GKH4A7FXMKRUNJY2LY7B5VCHHQVDSC64KOCVSTQJBNQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.930888403 (2h18m46.970461523s) [noise=true , address=77.90.31.15:4697 , piece_id=BV5JEJQFK36WR5SWQKPH2B5SUWHFQFUWHDG3ZNAWFHX6J63G7WAA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:11.061240724 (2h18m47.100813844s) [noise=true , address=77.90.31.18:4807 , piece_id=N7A3YJCCGHUG43CEPFFMAJNAWT34M3F7O3BOYEULM7GUKCZYMYHA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:12.512126805 (2h18m48.551699925s) [noise=true , address=45.91.116.13:4155 , piece_id=CKCFX75SIP63OM5WM2LUGMTS3PUEEJLZAA3F437T7SIUKGAVEPPQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:12.512155750 (2h18m48.55172887s) [seg_pos={0 0}, upload=2013]: Segment reached optimal threshold of 13 pieces. Sep 6 15:47:12.512396922 (2h18m48.551970042s) [noise=true , address=77.90.31.17:4770 , piece_id=V7NOD2DUP2Y4AMCBJMYJWZG2MRIO3WXS7GH6NX47FAHRLP7SPL6A, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45210->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 6 15:47:12.512444063 (2h18m48.552017183s) [noise=true , address=77.90.31.10:4518 , piece_id=RRCMSPFPOBAJERD7M4KK6RW5ODXZ2FQCZIOQSJ3M6WJ6CRTPUXZQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection Sep 6 15:47:12.512476094 (2h18m48.552049214s) [noise=true , address=77.90.31.17:4771 , piece_id=7DHNZZIUU5WY4YGK5SXZMFGENELXMY6UW62HINBQMMHHAJKWVBMQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:41736->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 6 15:47:12.512502023 (2h18m48.552075133s) [seg_pos={0 0}, upload=2013]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:47:12.512574802 (2h18m48.552147922s) [upload=2013]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:47:12.512593177 (2h18m48.552166307s) [upload=2013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:47:13.454815412 (2h18m49.494388533s) [upload=2013]: Done uploading segments... Sep 6 15:47:13.454834750 (2h18m49.49440787s) [upload=2013]: Upload finished. err: <nil> Duration: 14.562776766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.388298943s FAIL (2025-09-06 15:47:42 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.388326957s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X.XX] duration: 30.002137379s FAIL (2025-09-06 15:47:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:47:01.332418454 (2h18m37.371991574s) [upload=2013]: Starting upload Sep 6 15:47:01.332472658 (2h18m37.372045778s) [upload=2013]: Done starting upload Sep 6 15:47:01.332545006 (2h18m37.372118126s) [upload=2013]: Uploading segments... Sep 6 15:47:01.332656139 (2h18m37.372229259s) [upload=2013]: Got next segment. Inline: false Sep 6 15:47:01.332700954 (2h18m37.372274074s) [upload=2013]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:47:01.332706404 (2h18m37.372279524s) [upload=2013]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:47:03.864074672 (2h18m39.903647822s) [seg_pos={0 0}, upload=2013]: Begin upload segment... Sep 6 15:47:03.864137852 (2h18m39.903710972s) [seg_pos={0 0}, upload=2013]: Done begin upload segment. Sep 6 15:47:03.864168751 (2h18m39.903741871s) [noise=true , address=45.91.116.5:4298 , piece_id=TB4AUZC27GKH4A7FXMKRUNJY2LY7B5VCHHQVDSC64KOCVSTQJBNQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.864205001 (2h18m39.903778111s) [noise=true , address=45.91.116.13:4153 , piece_id=KFELQSQS7IFE7LTESU4MMF3EC4AKA2EPIKSYTLDLHA4YQ3WHK6CA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.864705579 (2h18m39.904278709s) [noise=true , address=77.90.31.12:4592 , piece_id=6DNDMYDDDXFNDHFKSXMXY6GK2FL7ZPM44HFUK6UWAI5VYLF3MIJA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.865171360 (2h18m39.9047445s) [noise=true , address=45.91.116.13:4155 , piece_id=CKCFX75SIP63OM5WM2LUGMTS3PUEEJLZAA3F437T7SIUKGAVEPPQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.865529185 (2h18m39.905102305s) [noise=true , address=77.90.31.17:4770 , piece_id=V7NOD2DUP2Y4AMCBJMYJWZG2MRIO3WXS7GH6NX47FAHRLP7SPL6A, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866349194 (2h18m39.905922314s) [noise=true , address=77.90.31.18:4807 , piece_id=N7A3YJCCGHUG43CEPFFMAJNAWT34M3F7O3BOYEULM7GUKCZYMYHA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866634881 (2h18m39.906208001s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZTDSLWTJCO7RZJ2J3YI67I6V37PXTA5QADYP6WGRYWFT7DSTR7Q, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.866889548 (2h18m39.906462668s) [noise=true , address=45.91.116.11:4119 , piece_id=KJSC7TTU4PYDY464PLGOJKY6EKXWVMEU2B4TTA5GQTEMB453RMTQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867192718 (2h18m39.906765838s) [noise=true , address=77.90.31.14:4663 , piece_id=7WMBOWSGQJEI2NMRPLVZLU4NPQRLXS4RDM5LUF2D3F6DE373CDCA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867465400 (2h18m39.90703852s) [noise=true , address=77.90.31.18:4805 , piece_id=F3ULLBFHPRJICXJYXFA3NWZ42QIXTYNUWP37CDKF7OTOSQVZHKTA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.867658780 (2h18m39.9072319s) [noise=true , address=77.90.31.15:4697 , piece_id=BV5JEJQFK36WR5SWQKPH2B5SUWHFQFUWHDG3ZNAWFHX6J63G7WAA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868116847 (2h18m39.907689977s) [noise=true , address=77.90.31.13:4628 , piece_id=QOZLWJHDST3QWVGIQJNWUWFHPENRJBOWDHCOIXK25TIIVTDBIVFQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868459953 (2h18m39.908033073s) [noise=true , address=77.90.31.17:4771 , piece_id=7DHNZZIUU5WY4YGK5SXZMFGENELXMY6UW62HINBQMMHHAJKWVBMQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.868809081 (2h18m39.908382201s) [noise=true , address=77.90.31.13:4627 , piece_id=L5ICEF3FDFE2E5635AEXOKY43CRLMZIDLIVG7AY5GSO2M67NYZCQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.869168819 (2h18m39.908741939s) [noise=true , address=77.90.31.21:4879 , piece_id=PARXJAP4ZTYRCBNGSEMEFFQ4HL2H5DREFY4FTK77O3OUHJA4BJAA, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:03.869481828 (2h18m39.909054949s) [noise=true , address=77.90.31.10:4518 , piece_id=RRCMSPFPOBAJERD7M4KK6RW5ODXZ2FQCZIOQSJ3M6WJ6CRTPUXZQ, seg_pos={0 0}, upload=2013]: Uploading piece... Sep 6 15:47:07.678956418 (2h18m43.718529538s) [upload=2013]: Next returned nil segment Sep 6 15:47:07.678972889 (2h18m43.718546009s) [upload=2013]: Waiting for error group managing segments... Sep 6 15:47:09.112289519 (2h18m45.151862639s) [noise=true , address=45.91.116.13:4153 , piece_id=KFELQSQS7IFE7LTESU4MMF3EC4AKA2EPIKSYTLDLHA4YQ3WHK6CA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.354792813 (2h18m45.394365933s) [noise=true , address=77.90.31.18:4805 , piece_id=F3ULLBFHPRJICXJYXFA3NWZ42QIXTYNUWP37CDKF7OTOSQVZHKTA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.406791473 (2h18m45.446364593s) [noise=true , address=77.90.31.12:4592 , piece_id=6DNDMYDDDXFNDHFKSXMXY6GK2FL7ZPM44HFUK6UWAI5VYLF3MIJA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.558598698 (2h18m45.598171818s) [noise=true , address=77.90.31.14:4663 , piece_id=7WMBOWSGQJEI2NMRPLVZLU4NPQRLXS4RDM5LUF2D3F6DE373CDCA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.578082551 (2h18m45.617655671s) [noise=true , address=77.90.31.13:4628 , piece_id=QOZLWJHDST3QWVGIQJNWUWFHPENRJBOWDHCOIXK25TIIVTDBIVFQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.800647251 (2h18m45.840220371s) [noise=true , address=77.90.31.13:4627 , piece_id=L5ICEF3FDFE2E5635AEXOKY43CRLMZIDLIVG7AY5GSO2M67NYZCQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.922688218 (2h18m45.962261338s) [noise=true , address=45.91.116.7:4010 , piece_id=4ZTDSLWTJCO7RZJ2J3YI67I6V37PXTA5QADYP6WGRYWFT7DSTR7Q, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:09.930649503 (2h18m45.970222613s) [noise=true , address=77.90.31.21:4879 , piece_id=PARXJAP4ZTYRCBNGSEMEFFQ4HL2H5DREFY4FTK77O3OUHJA4BJAA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.728726764 (2h18m46.768299884s) [noise=true , address=45.91.116.11:4119 , piece_id=KJSC7TTU4PYDY464PLGOJKY6EKXWVMEU2B4TTA5GQTEMB453RMTQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.923455901 (2h18m46.963029021s) [noise=true , address=45.91.116.5:4298 , piece_id=TB4AUZC27GKH4A7FXMKRUNJY2LY7B5VCHHQVDSC64KOCVSTQJBNQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:10.930888403 (2h18m46.970461523s) [noise=true , address=77.90.31.15:4697 , piece_id=BV5JEJQFK36WR5SWQKPH2B5SUWHFQFUWHDG3ZNAWFHX6J63G7WAA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:11.061240724 (2h18m47.100813844s) [noise=true , address=77.90.31.18:4807 , piece_id=N7A3YJCCGHUG43CEPFFMAJNAWT34M3F7O3BOYEULM7GUKCZYMYHA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:12.512126805 (2h18m48.551699925s) [noise=true , address=45.91.116.13:4155 , piece_id=CKCFX75SIP63OM5WM2LUGMTS3PUEEJLZAA3F437T7SIUKGAVEPPQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Sep 6 15:47:12.512155750 (2h18m48.55172887s) [seg_pos={0 0}, upload=2013]: Segment reached optimal threshold of 13 pieces. Sep 6 15:47:12.512396922 (2h18m48.551970042s) [noise=true , address=77.90.31.17:4770 , piece_id=V7NOD2DUP2Y4AMCBJMYJWZG2MRIO3WXS7GH6NX47FAHRLP7SPL6A, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:45210->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 6 15:47:12.512444063 (2h18m48.552017183s) [noise=true , address=77.90.31.10:4518 , piece_id=RRCMSPFPOBAJERD7M4KK6RW5ODXZ2FQCZIOQSJ3M6WJ6CRTPUXZQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56624->77.90.31.10:4518: use of closed network connection Sep 6 15:47:12.512476094 (2h18m48.552049214s) [noise=true , address=77.90.31.17:4771 , piece_id=7DHNZZIUU5WY4YGK5SXZMFGENELXMY6UW62HINBQMMHHAJKWVBMQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:41736->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 6 15:47:12.512502023 (2h18m48.552075133s) [seg_pos={0 0}, upload=2013]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:47:12.512574802 (2h18m48.552147922s) [upload=2013]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:47:12.512593177 (2h18m48.552166307s) [upload=2013]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:47:13.454815412 (2h18m49.494388533s) [upload=2013]: Done uploading segments... Sep 6 15:47:13.454834750 (2h18m49.49440787s) [upload=2013]: Upload finished. err: <nil> Duration: 30.002147158s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..............X] duration: 30.000806294s FAIL (2025-09-06 15:47:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000826632s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001151429s FAIL (2025-09-06 15:47:03 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001174783s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X.X] duration: 30.002534667s FAIL (2025-09-06 15:46:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:46:29.603569813 (2h18m5.643142933s) [upload=2006]: Starting upload Sep 6 15:46:29.603626090 (2h18m5.64319921s) [upload=2006]: Done starting upload Sep 6 15:46:29.603678290 (2h18m5.64325141s) [upload=2006]: Uploading segments... Sep 6 15:46:29.603806205 (2h18m5.643379326s) [upload=2006]: Got next segment. Inline: false Sep 6 15:46:29.603842565 (2h18m5.643415685s) [upload=2006]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:46:29.603847334 (2h18m5.643420454s) [upload=2006]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:46:32.172425703 (2h18m8.211998823s) [seg_pos={0 0}, upload=2006]: Begin upload segment... Sep 6 15:46:32.172472092 (2h18m8.212045212s) [seg_pos={0 0}, upload=2006]: Done begin upload segment. Sep 6 15:46:32.172493983 (2h18m8.212067103s) [noise=true , address=45.91.116.8:4048 , piece_id=PKYDKI3XIBLWLFRTQSGXLRAEKYWZV2LQI4UXGA522VVTXITN5T7A, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.172529742 (2h18m8.212102862s) [noise=true , address=77.90.31.18:4807 , piece_id=Y34PBAKMBBLS2PXZLHVVEY47UDDAJ4GWSM43IXUSU4NTCV5J6BQA, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.172887647 (2h18m8.212460757s) [noise=true , address=45.91.116.14:4191 , piece_id=BJKAY3IDNTEL6OVKWYRDD4N5K5PQ3CMVLMTN5FJQSZRKRLB4CNEA, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.173416098 (2h18m8.212989218s) [noise=true , address=45.91.116.10:4081 , piece_id=K7W536YH7AMNPNHOF6WQ7GORHLGTLAZ2GLBRG3RJXXO5OBX4LOVQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.173464901 (2h18m8.213038041s) [noise=true , address=77.90.31.14:4662 , piece_id=3YSSDY76XQ352M3EUD5GCJ5MTI3J6AIPEMPNHAKOJXZZLNIGV66Q, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.173786878 (2h18m8.213359998s) [noise=true , address=77.90.31.19:4843 , piece_id=O7XXIPE6GOT6REWEQLZO2Z44DIIC7PWBDABRA3LXM54VHZCDHEZQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.173992591 (2h18m8.213565711s) [noise=true , address=45.91.116.5:4299 , piece_id=FFP7FQQUGK53LO53MQN6OLTH3L322UF3Q75ZGTGGBE37LTU4JMSQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174060411 (2h18m8.213633531s) [noise=true , address=77.90.31.21:4879 , piece_id=BV6RZGPU7HXCTSAAPDFHEFILDX47KXKQORJNCGV2EEYQPWHXEKPQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174287555 (2h18m8.213860675s) [noise=true , address=45.91.116.6:4974 , piece_id=R3OUFXRGWB2QBZJGPVBMCVRO67J7BPSQZ6ZORMFMZKEIALPNPWTQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174307854 (2h18m8.213880975s) [noise=true , address=45.91.116.7:4009 , piece_id=SPASL74LEDSFMHKQH4KNIOJIXWRPWHJXO4TO32M6C62M76CTOXLA, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174332802 (2h18m8.213905922s) [noise=true , address=77.90.31.13:4627 , piece_id=CLEK6WJCYY5D7CP6EJ75QUWMANVD4L6ZNYC4AFVJWASV2OHA2IFQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174550278 (2h18m8.214123399s) [noise=true , address=77.90.31.18:4805 , piece_id=FFQIEQWKVX7J6R7KAXMNUURJW7NDFW7TSXHN26TDAVK6G7UFXGIA, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174554747 (2h18m8.214127867s) [noise=true , address=77.90.31.12:4590 , piece_id=37BZHT53OXXDYDYESCQ24DQ2TNJ5E57627RZ43O7IVJ7MOBQOJQA, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.174790968 (2h18m8.214364088s) [noise=true , address=45.91.116.9:4263 , piece_id=SGK43635W4RWMEVKHTLWSGPRSFXBNX6EUG37LY5OQEUMTI7BDD2A, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.175074992 (2h18m8.214648112s) [noise=true , address=77.90.31.14:4661 , piece_id=PQV3YB6HWCVW364DRQUQ2ZC2ZCDCBI3AVBAL23KWWQNRYQ2Y7YLQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:32.175327325 (2h18m8.214900445s) [noise=true , address=45.91.116.5:4298 , piece_id=MM7BCTLMQXXYX3JF4YPFFZ2BRHWWMPX7NWIC2HQX72QEXQH2Q7YQ, seg_pos={0 0}, upload=2006]: Uploading piece... Sep 6 15:46:36.095691354 (2h18m12.135264474s) [upload=2006]: Next returned nil segment Sep 6 15:46:36.095708927 (2h18m12.135282047s) [upload=2006]: Waiting for error group managing segments... Sep 6 15:46:36.895602214 (2h18m12.935175334s) [noise=true , address=77.90.31.19:4843 , piece_id=O7XXIPE6GOT6REWEQLZO2Z44DIIC7PWBDABRA3LXM54VHZCDHEZQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:37.306160063 (2h18m13.345733183s) [noise=true , address=77.90.31.14:4661 , piece_id=PQV3YB6HWCVW364DRQUQ2ZC2ZCDCBI3AVBAL23KWWQNRYQ2Y7YLQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:37.773486737 (2h18m13.813059858s) [noise=true , address=77.90.31.12:4590 , piece_id=37BZHT53OXXDYDYESCQ24DQ2TNJ5E57627RZ43O7IVJ7MOBQOJQA, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:37.934487825 (2h18m13.974060945s) [noise=true , address=77.90.31.14:4662 , piece_id=3YSSDY76XQ352M3EUD5GCJ5MTI3J6AIPEMPNHAKOJXZZLNIGV66Q, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:37.970393632 (2h18m14.009966752s) [noise=true , address=77.90.31.13:4627 , piece_id=CLEK6WJCYY5D7CP6EJ75QUWMANVD4L6ZNYC4AFVJWASV2OHA2IFQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:37.973981117 (2h18m14.013554237s) [noise=true , address=45.91.116.8:4048 , piece_id=PKYDKI3XIBLWLFRTQSGXLRAEKYWZV2LQI4UXGA522VVTXITN5T7A, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:38.207606362 (2h18m14.247179482s) [noise=true , address=45.91.116.10:4081 , piece_id=K7W536YH7AMNPNHOF6WQ7GORHLGTLAZ2GLBRG3RJXXO5OBX4LOVQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:38.441910746 (2h18m14.481483866s) [noise=true , address=77.90.31.18:4807 , piece_id=Y34PBAKMBBLS2PXZLHVVEY47UDDAJ4GWSM43IXUSU4NTCV5J6BQA, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:38.910085364 (2h18m14.949658484s) [noise=true , address=45.91.116.5:4298 , piece_id=MM7BCTLMQXXYX3JF4YPFFZ2BRHWWMPX7NWIC2HQX72QEXQH2Q7YQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:38.989474115 (2h18m15.029047235s) [noise=true , address=45.91.116.5:4299 , piece_id=FFP7FQQUGK53LO53MQN6OLTH3L322UF3Q75ZGTGGBE37LTU4JMSQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:39.875687205 (2h18m15.915260325s) [noise=true , address=45.91.116.6:4974 , piece_id=R3OUFXRGWB2QBZJGPVBMCVRO67J7BPSQZ6ZORMFMZKEIALPNPWTQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:40.191043132 (2h18m16.230616252s) [noise=true , address=45.91.116.14:4191 , piece_id=BJKAY3IDNTEL6OVKWYRDD4N5K5PQ3CMVLMTN5FJQSZRKRLB4CNEA, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:47.857916154 (2h18m23.897489274s) [noise=true , address=77.90.31.18:4805 , piece_id=FFQIEQWKVX7J6R7KAXMNUURJW7NDFW7TSXHN26TDAVK6G7UFXGIA, seg_pos={0 0}, upload=2006]: Done uploading piece. err: <nil> Sep 6 15:46:47.857942224 (2h18m23.897515344s) [seg_pos={0 0}, upload=2006]: Segment reached optimal threshold of 13 pieces. Sep 6 15:46:47.858041473 (2h18m23.897614583s) [noise=true , address=45.91.116.7:4009 , piece_id=SPASL74LEDSFMHKQH4KNIOJIXWRPWHJXO4TO32M6C62M76CTOXLA, seg_pos={0 0}, upload=2006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Sep 6 15:46:47.858079005 (2h18m23.897652126s) [noise=true , address=45.91.116.9:4263 , piece_id=SGK43635W4RWMEVKHTLWSGPRSFXBNX6EUG37LY5OQEUMTI7BDD2A, seg_pos={0 0}, upload=2006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Sep 6 15:46:47.858134371 (2h18m23.897707481s) [noise=true , address=77.90.31.21:4879 , piece_id=BV6RZGPU7HXCTSAAPDFHEFILDX47KXKQORJNCGV2EEYQPWHXEKPQ, seg_pos={0 0}, upload=2006]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 188.166.225.170:52368->77.90.31.21:4879: use of closed network connection; write tcp 188.166.225.170:52368->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52368->77.90.31.21:4879: use of closed network connection Sep 6 15:46:47.858154520 (2h18m23.89772764s) [seg_pos={0 0}, upload=2006]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:46:47.858219364 (2h18m23.897792474s) [upload=2006]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:46:47.858224153 (2h18m23.897797273s) [upload=2006]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:46:48.754119390 (2h18m24.79369251s) [upload=2006]: Done uploading segments... Sep 6 15:46:48.754138407 (2h18m24.793711527s) [upload=2006]: Upload finished. err: <nil> Duration: 30.002548322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X.] duration: 27.3173672s FAIL (2025-09-06 15:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:45:01.633702232 (2h16m37.673275362s) [upload=1985]: Starting upload Sep 6 15:45:01.633789810 (2h16m37.67336294s) [upload=1985]: Done starting upload Sep 6 15:45:01.633909949 (2h16m37.673483069s) [upload=1985]: Uploading segments... Sep 6 15:45:01.634055569 (2h16m37.673628689s) [upload=1985]: Got next segment. Inline: false Sep 6 15:45:01.634095304 (2h16m37.673668424s) [upload=1985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:45:01.634102057 (2h16m37.673675187s) [upload=1985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:45:02.143745848 (2h16m38.183318978s) [seg_pos={0 0}, upload=1985]: Begin upload segment... Sep 6 15:45:02.143812385 (2h16m38.183385515s) [seg_pos={0 0}, upload=1985]: Done begin upload segment. Sep 6 15:45:02.143851200 (2h16m38.18342432s) [noise=true , address=77.90.31.18:4807 , piece_id=H4GDPMGM3LZWP3UVXXGG2NNLKVX6TQ5LRGLOTYJJAFJJSNIFQVPA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.143888080 (2h16m38.18346122s) [noise=true , address=45.91.116.4:4937 , piece_id=EOX2BKMQZPKV7RTNIOL7DQ7EGJUHP2XBUDOKKFM3437IZXF6U4GQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.144336407 (2h16m38.183909538s) [noise=true , address=45.91.116.13:4153 , piece_id=GSY67D4TASPUVTLERGA7JXYVJHPWFUTO4CVVYXNW7MFRYMBEMINA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.144548984 (2h16m38.184122115s) [noise=true , address=45.91.116.8:4048 , piece_id=6ZUHHASASZPYPTIYKNGWVMASWL5RG7K6ZTXFYFOOGXL5NGNNHDDA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.144761321 (2h16m38.184334441s) [noise=true , address=45.91.116.11:4119 , piece_id=6V6OS3XFU6LG6SH2MHRRQGZLJ6MBY7M3WGDYVATNNTE7QMMNR27A, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.145016018 (2h16m38.184589158s) [noise=true , address=77.90.31.15:4698 , piece_id=OY3EUI4WHTEKKD75YGNCLPIF7S3KR62KRLYCDCKCYQBGZ5NXWCOA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.145207404 (2h16m38.184780524s) [noise=true , address=77.90.31.19:4842 , piece_id=RWJUK5G2RW7MFZO5T3YJZZJRGXOWUVVRNU6PLRFUGXA6PEYRM4LQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.145619994 (2h16m38.185193124s) [noise=true , address=77.90.31.21:4878 , piece_id=B2JD6VL76IF3XP7J4QUT57RL434BY2NBZAMKO4UYPDUBC53GEHLA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.146112055 (2h16m38.185685185s) [noise=true , address=77.90.31.13:4626 , piece_id=SL66SC6ZTC324Z72DH4L63A6QX7HU4FJYWZGZLLHD3LTIEYWHZWQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.146532390 (2h16m38.18610552s) [noise=true , address=45.91.116.6:4974 , piece_id=BTWZQUQRLU52BLDYCWMO65WDH2QBMUPGK3RGIF7PIOWHKLEIVCZQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.146899472 (2h16m38.186472602s) [noise=true , address=77.90.31.15:4699 , piece_id=YQZRDHKOC3BIJKQ4J6RLNIUGBXT23DXF5X3EC6M7R2JOHOD2PC3A, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.147319385 (2h16m38.186892505s) [noise=true , address=45.91.116.13:4155 , piece_id=NPNYLLJSECXF6U2SR7GGW5RR6MAIULTBOP37XNJ4VTNUCRGUR43A, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.147728528 (2h16m38.187301658s) [noise=true , address=77.90.31.16:4735 , piece_id=YQETWRXHJC5EWDWRJTJLYK62KM2DSN52DVNF54HHG5SCJRRFOCGQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.147833740 (2h16m38.18740687s) [noise=true , address=77.90.31.18:4805 , piece_id=4WOLSS4TX3MTJVJXKQCYHU2RREHV5LGGOATBSL7CAENQFGSK6ROQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.148336432 (2h16m38.187909562s) [noise=true , address=77.90.31.17:4771 , piece_id=2ZFMY5TRSHLBKIQ4ZQ7QG4SHGNMCZNFL6V5HINGY5RNFYCNVW2HQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.149243478 (2h16m38.188816608s) [noise=true , address=45.91.116.7:4010 , piece_id=XCFUDOFII47BJSKMVHIDMSMAL2QK7XAEK43ZOMSDCN3MO37PWTUQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:03.979888804 (2h16m40.019461934s) [upload=1985]: Next returned nil segment Sep 6 15:45:03.979907819 (2h16m40.019480939s) [upload=1985]: Waiting for error group managing segments... Sep 6 15:45:04.306326065 (2h16m40.345899185s) [noise=true , address=77.90.31.13:4626 , piece_id=SL66SC6ZTC324Z72DH4L63A6QX7HU4FJYWZGZLLHD3LTIEYWHZWQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.350788314 (2h16m40.390361444s) [noise=true , address=77.90.31.19:4842 , piece_id=RWJUK5G2RW7MFZO5T3YJZZJRGXOWUVVRNU6PLRFUGXA6PEYRM4LQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.366778432 (2h16m40.406351552s) [noise=true , address=45.91.116.7:4010 , piece_id=XCFUDOFII47BJSKMVHIDMSMAL2QK7XAEK43ZOMSDCN3MO37PWTUQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.372802569 (2h16m40.412375689s) [noise=true , address=45.91.116.4:4937 , piece_id=EOX2BKMQZPKV7RTNIOL7DQ7EGJUHP2XBUDOKKFM3437IZXF6U4GQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.404890199 (2h16m40.444463319s) [noise=true , address=45.91.116.13:4153 , piece_id=GSY67D4TASPUVTLERGA7JXYVJHPWFUTO4CVVYXNW7MFRYMBEMINA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.417846751 (2h16m40.457419871s) [noise=true , address=77.90.31.15:4698 , piece_id=OY3EUI4WHTEKKD75YGNCLPIF7S3KR62KRLYCDCKCYQBGZ5NXWCOA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.450918094 (2h16m40.490491214s) [noise=true , address=45.91.116.11:4119 , piece_id=6V6OS3XFU6LG6SH2MHRRQGZLJ6MBY7M3WGDYVATNNTE7QMMNR27A, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.467171625 (2h16m40.506744745s) [noise=true , address=77.90.31.18:4805 , piece_id=4WOLSS4TX3MTJVJXKQCYHU2RREHV5LGGOATBSL7CAENQFGSK6ROQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.467237161 (2h16m40.506810281s) [noise=true , address=45.91.116.13:4155 , piece_id=NPNYLLJSECXF6U2SR7GGW5RR6MAIULTBOP37XNJ4VTNUCRGUR43A, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.467609062 (2h16m40.507182182s) [noise=true , address=77.90.31.16:4735 , piece_id=YQETWRXHJC5EWDWRJTJLYK62KM2DSN52DVNF54HHG5SCJRRFOCGQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.482301016 (2h16m40.521874146s) [noise=true , address=77.90.31.18:4807 , piece_id=H4GDPMGM3LZWP3UVXXGG2NNLKVX6TQ5LRGLOTYJJAFJJSNIFQVPA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.541159632 (2h16m40.580732752s) [noise=true , address=77.90.31.17:4771 , piece_id=2ZFMY5TRSHLBKIQ4ZQ7QG4SHGNMCZNFL6V5HINGY5RNFYCNVW2HQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.613763350 (2h16m40.65333647s) [noise=true , address=77.90.31.21:4878 , piece_id=B2JD6VL76IF3XP7J4QUT57RL434BY2NBZAMKO4UYPDUBC53GEHLA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.613830479 (2h16m40.653403599s) [seg_pos={0 0}, upload=1985]: Segment reached optimal threshold of 13 pieces. Sep 6 15:45:04.614125744 (2h16m40.653698875s) [noise=true , address=77.90.31.15:4699 , piece_id=YQZRDHKOC3BIJKQ4J6RLNIUGBXT23DXF5X3EC6M7R2JOHOD2PC3A, seg_pos={0 0}, upload=1985]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:58784->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:58784->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58784->77.90.31.15:4699: use of closed network connection Sep 6 15:45:04.614184607 (2h16m40.653757727s) [noise=true , address=45.91.116.6:4974 , piece_id=BTWZQUQRLU52BLDYCWMO65WDH2QBMUPGK3RGIF7PIOWHKLEIVCZQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:33762->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:33762->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33762->45.91.116.6:4974: use of closed network connection Sep 6 15:45:04.614198503 (2h16m40.653771623s) [noise=true , address=45.91.116.8:4048 , piece_id=6ZUHHASASZPYPTIYKNGWVMASWL5RG7K6ZTXFYFOOGXL5NGNNHDDA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Sep 6 15:45:04.614246495 (2h16m40.653819615s) [seg_pos={0 0}, upload=1985]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:45:04.614365753 (2h16m40.653938883s) [upload=1985]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:45:04.614371334 (2h16m40.653944454s) [upload=1985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:45:04.968113925 (2h16m41.007687045s) [upload=1985]: Done uploading segments... Sep 6 15:45:04.968134003 (2h16m41.007707124s) [upload=1985]: Upload finished. err: <nil> Duration: 27.317395534s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 9.34551126s FAIL (2025-09-06 15:46:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.345531228s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...X] duration: 30.001897341s FAIL (2025-09-06 15:45:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:45:01.633702232 (2h16m37.673275362s) [upload=1985]: Starting upload Sep 6 15:45:01.633789810 (2h16m37.67336294s) [upload=1985]: Done starting upload Sep 6 15:45:01.633909949 (2h16m37.673483069s) [upload=1985]: Uploading segments... Sep 6 15:45:01.634055569 (2h16m37.673628689s) [upload=1985]: Got next segment. Inline: false Sep 6 15:45:01.634095304 (2h16m37.673668424s) [upload=1985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:45:01.634102057 (2h16m37.673675187s) [upload=1985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:45:02.143745848 (2h16m38.183318978s) [seg_pos={0 0}, upload=1985]: Begin upload segment... Sep 6 15:45:02.143812385 (2h16m38.183385515s) [seg_pos={0 0}, upload=1985]: Done begin upload segment. Sep 6 15:45:02.143851200 (2h16m38.18342432s) [noise=true , address=77.90.31.18:4807 , piece_id=H4GDPMGM3LZWP3UVXXGG2NNLKVX6TQ5LRGLOTYJJAFJJSNIFQVPA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.143888080 (2h16m38.18346122s) [noise=true , address=45.91.116.4:4937 , piece_id=EOX2BKMQZPKV7RTNIOL7DQ7EGJUHP2XBUDOKKFM3437IZXF6U4GQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.144336407 (2h16m38.183909538s) [noise=true , address=45.91.116.13:4153 , piece_id=GSY67D4TASPUVTLERGA7JXYVJHPWFUTO4CVVYXNW7MFRYMBEMINA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.144548984 (2h16m38.184122115s) [noise=true , address=45.91.116.8:4048 , piece_id=6ZUHHASASZPYPTIYKNGWVMASWL5RG7K6ZTXFYFOOGXL5NGNNHDDA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.144761321 (2h16m38.184334441s) [noise=true , address=45.91.116.11:4119 , piece_id=6V6OS3XFU6LG6SH2MHRRQGZLJ6MBY7M3WGDYVATNNTE7QMMNR27A, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.145016018 (2h16m38.184589158s) [noise=true , address=77.90.31.15:4698 , piece_id=OY3EUI4WHTEKKD75YGNCLPIF7S3KR62KRLYCDCKCYQBGZ5NXWCOA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.145207404 (2h16m38.184780524s) [noise=true , address=77.90.31.19:4842 , piece_id=RWJUK5G2RW7MFZO5T3YJZZJRGXOWUVVRNU6PLRFUGXA6PEYRM4LQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.145619994 (2h16m38.185193124s) [noise=true , address=77.90.31.21:4878 , piece_id=B2JD6VL76IF3XP7J4QUT57RL434BY2NBZAMKO4UYPDUBC53GEHLA, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.146112055 (2h16m38.185685185s) [noise=true , address=77.90.31.13:4626 , piece_id=SL66SC6ZTC324Z72DH4L63A6QX7HU4FJYWZGZLLHD3LTIEYWHZWQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.146532390 (2h16m38.18610552s) [noise=true , address=45.91.116.6:4974 , piece_id=BTWZQUQRLU52BLDYCWMO65WDH2QBMUPGK3RGIF7PIOWHKLEIVCZQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.146899472 (2h16m38.186472602s) [noise=true , address=77.90.31.15:4699 , piece_id=YQZRDHKOC3BIJKQ4J6RLNIUGBXT23DXF5X3EC6M7R2JOHOD2PC3A, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.147319385 (2h16m38.186892505s) [noise=true , address=45.91.116.13:4155 , piece_id=NPNYLLJSECXF6U2SR7GGW5RR6MAIULTBOP37XNJ4VTNUCRGUR43A, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.147728528 (2h16m38.187301658s) [noise=true , address=77.90.31.16:4735 , piece_id=YQETWRXHJC5EWDWRJTJLYK62KM2DSN52DVNF54HHG5SCJRRFOCGQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.147833740 (2h16m38.18740687s) [noise=true , address=77.90.31.18:4805 , piece_id=4WOLSS4TX3MTJVJXKQCYHU2RREHV5LGGOATBSL7CAENQFGSK6ROQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.148336432 (2h16m38.187909562s) [noise=true , address=77.90.31.17:4771 , piece_id=2ZFMY5TRSHLBKIQ4ZQ7QG4SHGNMCZNFL6V5HINGY5RNFYCNVW2HQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:02.149243478 (2h16m38.188816608s) [noise=true , address=45.91.116.7:4010 , piece_id=XCFUDOFII47BJSKMVHIDMSMAL2QK7XAEK43ZOMSDCN3MO37PWTUQ, seg_pos={0 0}, upload=1985]: Uploading piece... Sep 6 15:45:03.979888804 (2h16m40.019461934s) [upload=1985]: Next returned nil segment Sep 6 15:45:03.979907819 (2h16m40.019480939s) [upload=1985]: Waiting for error group managing segments... Sep 6 15:45:04.306326065 (2h16m40.345899185s) [noise=true , address=77.90.31.13:4626 , piece_id=SL66SC6ZTC324Z72DH4L63A6QX7HU4FJYWZGZLLHD3LTIEYWHZWQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.350788314 (2h16m40.390361444s) [noise=true , address=77.90.31.19:4842 , piece_id=RWJUK5G2RW7MFZO5T3YJZZJRGXOWUVVRNU6PLRFUGXA6PEYRM4LQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.366778432 (2h16m40.406351552s) [noise=true , address=45.91.116.7:4010 , piece_id=XCFUDOFII47BJSKMVHIDMSMAL2QK7XAEK43ZOMSDCN3MO37PWTUQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.372802569 (2h16m40.412375689s) [noise=true , address=45.91.116.4:4937 , piece_id=EOX2BKMQZPKV7RTNIOL7DQ7EGJUHP2XBUDOKKFM3437IZXF6U4GQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.404890199 (2h16m40.444463319s) [noise=true , address=45.91.116.13:4153 , piece_id=GSY67D4TASPUVTLERGA7JXYVJHPWFUTO4CVVYXNW7MFRYMBEMINA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.417846751 (2h16m40.457419871s) [noise=true , address=77.90.31.15:4698 , piece_id=OY3EUI4WHTEKKD75YGNCLPIF7S3KR62KRLYCDCKCYQBGZ5NXWCOA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.450918094 (2h16m40.490491214s) [noise=true , address=45.91.116.11:4119 , piece_id=6V6OS3XFU6LG6SH2MHRRQGZLJ6MBY7M3WGDYVATNNTE7QMMNR27A, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.467171625 (2h16m40.506744745s) [noise=true , address=77.90.31.18:4805 , piece_id=4WOLSS4TX3MTJVJXKQCYHU2RREHV5LGGOATBSL7CAENQFGSK6ROQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.467237161 (2h16m40.506810281s) [noise=true , address=45.91.116.13:4155 , piece_id=NPNYLLJSECXF6U2SR7GGW5RR6MAIULTBOP37XNJ4VTNUCRGUR43A, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.467609062 (2h16m40.507182182s) [noise=true , address=77.90.31.16:4735 , piece_id=YQETWRXHJC5EWDWRJTJLYK62KM2DSN52DVNF54HHG5SCJRRFOCGQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.482301016 (2h16m40.521874146s) [noise=true , address=77.90.31.18:4807 , piece_id=H4GDPMGM3LZWP3UVXXGG2NNLKVX6TQ5LRGLOTYJJAFJJSNIFQVPA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.541159632 (2h16m40.580732752s) [noise=true , address=77.90.31.17:4771 , piece_id=2ZFMY5TRSHLBKIQ4ZQ7QG4SHGNMCZNFL6V5HINGY5RNFYCNVW2HQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.613763350 (2h16m40.65333647s) [noise=true , address=77.90.31.21:4878 , piece_id=B2JD6VL76IF3XP7J4QUT57RL434BY2NBZAMKO4UYPDUBC53GEHLA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: <nil> Sep 6 15:45:04.613830479 (2h16m40.653403599s) [seg_pos={0 0}, upload=1985]: Segment reached optimal threshold of 13 pieces. Sep 6 15:45:04.614125744 (2h16m40.653698875s) [noise=true , address=77.90.31.15:4699 , piece_id=YQZRDHKOC3BIJKQ4J6RLNIUGBXT23DXF5X3EC6M7R2JOHOD2PC3A, seg_pos={0 0}, upload=1985]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: write tcp 188.166.225.170:58784->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:58784->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58784->77.90.31.15:4699: use of closed network connection Sep 6 15:45:04.614184607 (2h16m40.653757727s) [noise=true , address=45.91.116.6:4974 , piece_id=BTWZQUQRLU52BLDYCWMO65WDH2QBMUPGK3RGIF7PIOWHKLEIVCZQ, seg_pos={0 0}, upload=1985]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:33762->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:33762->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33762->45.91.116.6:4974: use of closed network connection Sep 6 15:45:04.614198503 (2h16m40.653771623s) [noise=true , address=45.91.116.8:4048 , piece_id=6ZUHHASASZPYPTIYKNGWVMASWL5RG7K6ZTXFYFOOGXL5NGNNHDDA, seg_pos={0 0}, upload=1985]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Sep 6 15:45:04.614246495 (2h16m40.653819615s) [seg_pos={0 0}, upload=1985]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:45:04.614365753 (2h16m40.653938883s) [upload=1985]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:45:04.614371334 (2h16m40.653944454s) [upload=1985]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:45:04.968113925 (2h16m41.007687045s) [upload=1985]: Done uploading segments... Sep 6 15:45:04.968134003 (2h16m41.007707124s) [upload=1985]: Upload finished. err: <nil> Duration: 30.001920253s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.284495911s FAIL (2025-09-06 15:45:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.284520628s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X...] duration: 18.309833623s FAIL (2025-09-06 15:44:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:42:33.126606730 (2h14m9.16617985s) [upload=1948]: Starting upload Sep 6 15:42:33.126653710 (2h14m9.16622683s) [upload=1948]: Done starting upload Sep 6 15:42:33.126812484 (2h14m9.166385605s) [upload=1948]: Uploading segments... Sep 6 15:42:33.126870044 (2h14m9.166443155s) [upload=1948]: Got next segment. Inline: false Sep 6 15:42:33.126897647 (2h14m9.166470767s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:42:33.126902436 (2h14m9.166475556s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:42:33.410793315 (2h14m9.450366475s) [seg_pos={0 0}, upload=1948]: Begin upload segment... Sep 6 15:42:33.410856756 (2h14m9.450429916s) [seg_pos={0 0}, upload=1948]: Done begin upload segment. Sep 6 15:42:33.410882044 (2h14m9.450455205s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410894068 (2h14m9.450467188s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410952890 (2h14m9.45052601s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411791985 (2h14m9.451365135s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411842222 (2h14m9.451415382s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412379730 (2h14m9.45195285s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412426190 (2h14m9.45199931s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412989457 (2h14m9.452562587s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413370807 (2h14m9.452943927s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413742228 (2h14m9.453315348s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414120341 (2h14m9.453693462s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414494016 (2h14m9.454067146s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414819719 (2h14m9.454392839s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415143870 (2h14m9.45471699s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415653796 (2h14m9.455226916s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415993605 (2h14m9.455566756s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:35.252940940 (2h14m11.29251406s) [upload=1948]: Next returned nil segment Sep 6 15:42:35.252962000 (2h14m11.29253512s) [upload=1948]: Waiting for error group managing segments... Sep 6 15:42:35.559652977 (2h14m11.599226097s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.596991304 (2h14m11.636564425s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.604569054 (2h14m11.644142174s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.625211023 (2h14m11.664784153s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.685780665 (2h14m11.725353785s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.708016543 (2h14m11.747589663s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.784454438 (2h14m11.824027558s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.799625067 (2h14m11.839198187s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.834792140 (2h14m11.87436526s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.896840079 (2h14m11.936413199s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.908786489 (2h14m11.94835961s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.997513589 (2h14m12.037086719s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472040463 (2h14m12.511613583s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472066212 (2h14m12.511639332s) [seg_pos={0 0}, upload=1948]: Segment reached optimal threshold of 13 pieces. Sep 6 15:42:36.472174839 (2h14m12.51174796s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472215037 (2h14m12.511788157s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472233703 (2h14m12.511806823s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472249182 (2h14m12.511822302s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:42:36.472336128 (2h14m12.511909248s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:42:36.472340708 (2h14m12.511913828s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:42:36.839281552 (2h14m12.878854683s) [upload=1948]: Done uploading segments... Sep 6 15:42:36.839305619 (2h14m12.878878789s) [upload=1948]: Upload finished. err: <nil> Duration: 18.30985753s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.141711165s FAIL (2025-09-06 15:44:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.141728469s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X..] duration: 6.706095776s FAIL (2025-09-06 15:44:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:42:33.126606730 (2h14m9.16617985s) [upload=1948]: Starting upload Sep 6 15:42:33.126653710 (2h14m9.16622683s) [upload=1948]: Done starting upload Sep 6 15:42:33.126812484 (2h14m9.166385605s) [upload=1948]: Uploading segments... Sep 6 15:42:33.126870044 (2h14m9.166443155s) [upload=1948]: Got next segment. Inline: false Sep 6 15:42:33.126897647 (2h14m9.166470767s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:42:33.126902436 (2h14m9.166475556s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:42:33.410793315 (2h14m9.450366475s) [seg_pos={0 0}, upload=1948]: Begin upload segment... Sep 6 15:42:33.410856756 (2h14m9.450429916s) [seg_pos={0 0}, upload=1948]: Done begin upload segment. Sep 6 15:42:33.410882044 (2h14m9.450455205s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410894068 (2h14m9.450467188s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410952890 (2h14m9.45052601s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411791985 (2h14m9.451365135s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411842222 (2h14m9.451415382s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412379730 (2h14m9.45195285s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412426190 (2h14m9.45199931s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412989457 (2h14m9.452562587s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413370807 (2h14m9.452943927s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413742228 (2h14m9.453315348s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414120341 (2h14m9.453693462s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414494016 (2h14m9.454067146s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414819719 (2h14m9.454392839s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415143870 (2h14m9.45471699s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415653796 (2h14m9.455226916s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415993605 (2h14m9.455566756s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:35.252940940 (2h14m11.29251406s) [upload=1948]: Next returned nil segment Sep 6 15:42:35.252962000 (2h14m11.29253512s) [upload=1948]: Waiting for error group managing segments... Sep 6 15:42:35.559652977 (2h14m11.599226097s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.596991304 (2h14m11.636564425s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.604569054 (2h14m11.644142174s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.625211023 (2h14m11.664784153s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.685780665 (2h14m11.725353785s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.708016543 (2h14m11.747589663s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.784454438 (2h14m11.824027558s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.799625067 (2h14m11.839198187s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.834792140 (2h14m11.87436526s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.896840079 (2h14m11.936413199s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.908786489 (2h14m11.94835961s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.997513589 (2h14m12.037086719s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472040463 (2h14m12.511613583s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472066212 (2h14m12.511639332s) [seg_pos={0 0}, upload=1948]: Segment reached optimal threshold of 13 pieces. Sep 6 15:42:36.472174839 (2h14m12.51174796s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472215037 (2h14m12.511788157s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472233703 (2h14m12.511806823s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472249182 (2h14m12.511822302s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:42:36.472336128 (2h14m12.511909248s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:42:36.472340708 (2h14m12.511913828s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:42:36.839281552 (2h14m12.878854683s) [upload=1948]: Done uploading segments... Sep 6 15:42:36.839305619 (2h14m12.878878789s) [upload=1948]: Upload finished. err: <nil> Duration: 6.706126946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X.] duration: 8.318795861s FAIL (2025-09-06 15:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:42:33.126606730 (2h14m9.16617985s) [upload=1948]: Starting upload Sep 6 15:42:33.126653710 (2h14m9.16622683s) [upload=1948]: Done starting upload Sep 6 15:42:33.126812484 (2h14m9.166385605s) [upload=1948]: Uploading segments... Sep 6 15:42:33.126870044 (2h14m9.166443155s) [upload=1948]: Got next segment. Inline: false Sep 6 15:42:33.126897647 (2h14m9.166470767s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:42:33.126902436 (2h14m9.166475556s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:42:33.410793315 (2h14m9.450366475s) [seg_pos={0 0}, upload=1948]: Begin upload segment... Sep 6 15:42:33.410856756 (2h14m9.450429916s) [seg_pos={0 0}, upload=1948]: Done begin upload segment. Sep 6 15:42:33.410882044 (2h14m9.450455205s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410894068 (2h14m9.450467188s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410952890 (2h14m9.45052601s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411791985 (2h14m9.451365135s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411842222 (2h14m9.451415382s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412379730 (2h14m9.45195285s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412426190 (2h14m9.45199931s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412989457 (2h14m9.452562587s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413370807 (2h14m9.452943927s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413742228 (2h14m9.453315348s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414120341 (2h14m9.453693462s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414494016 (2h14m9.454067146s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414819719 (2h14m9.454392839s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415143870 (2h14m9.45471699s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415653796 (2h14m9.455226916s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415993605 (2h14m9.455566756s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:35.252940940 (2h14m11.29251406s) [upload=1948]: Next returned nil segment Sep 6 15:42:35.252962000 (2h14m11.29253512s) [upload=1948]: Waiting for error group managing segments... Sep 6 15:42:35.559652977 (2h14m11.599226097s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.596991304 (2h14m11.636564425s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.604569054 (2h14m11.644142174s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.625211023 (2h14m11.664784153s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.685780665 (2h14m11.725353785s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.708016543 (2h14m11.747589663s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.784454438 (2h14m11.824027558s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.799625067 (2h14m11.839198187s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.834792140 (2h14m11.87436526s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.896840079 (2h14m11.936413199s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.908786489 (2h14m11.94835961s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.997513589 (2h14m12.037086719s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472040463 (2h14m12.511613583s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472066212 (2h14m12.511639332s) [seg_pos={0 0}, upload=1948]: Segment reached optimal threshold of 13 pieces. Sep 6 15:42:36.472174839 (2h14m12.51174796s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472215037 (2h14m12.511788157s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472233703 (2h14m12.511806823s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472249182 (2h14m12.511822302s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:42:36.472336128 (2h14m12.511909248s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:42:36.472340708 (2h14m12.511913828s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:42:36.839281552 (2h14m12.878854683s) [upload=1948]: Done uploading segments... Sep 6 15:42:36.839305619 (2h14m12.878878789s) [upload=1948]: Upload finished. err: <nil> Duration: 8.318819597s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.760157817s FAIL (2025-09-06 15:43:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.76017985s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.X] duration: 30.002968434s FAIL (2025-09-06 15:43:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:42:33.126606730 (2h14m9.16617985s) [upload=1948]: Starting upload Sep 6 15:42:33.126653710 (2h14m9.16622683s) [upload=1948]: Done starting upload Sep 6 15:42:33.126812484 (2h14m9.166385605s) [upload=1948]: Uploading segments... Sep 6 15:42:33.126870044 (2h14m9.166443155s) [upload=1948]: Got next segment. Inline: false Sep 6 15:42:33.126897647 (2h14m9.166470767s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:42:33.126902436 (2h14m9.166475556s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:42:33.410793315 (2h14m9.450366475s) [seg_pos={0 0}, upload=1948]: Begin upload segment... Sep 6 15:42:33.410856756 (2h14m9.450429916s) [seg_pos={0 0}, upload=1948]: Done begin upload segment. Sep 6 15:42:33.410882044 (2h14m9.450455205s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410894068 (2h14m9.450467188s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.410952890 (2h14m9.45052601s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411791985 (2h14m9.451365135s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.411842222 (2h14m9.451415382s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412379730 (2h14m9.45195285s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412426190 (2h14m9.45199931s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.412989457 (2h14m9.452562587s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413370807 (2h14m9.452943927s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.413742228 (2h14m9.453315348s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414120341 (2h14m9.453693462s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414494016 (2h14m9.454067146s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.414819719 (2h14m9.454392839s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415143870 (2h14m9.45471699s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415653796 (2h14m9.455226916s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:33.415993605 (2h14m9.455566756s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Uploading piece... Sep 6 15:42:35.252940940 (2h14m11.29251406s) [upload=1948]: Next returned nil segment Sep 6 15:42:35.252962000 (2h14m11.29253512s) [upload=1948]: Waiting for error group managing segments... Sep 6 15:42:35.559652977 (2h14m11.599226097s) [noise=true , address=45.91.116.10:4083 , piece_id=XSEGXMJI43FXW7BXQQ4KKESD4QLDV2M6MCRZQLNB5XOPPIWGOEBA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.596991304 (2h14m11.636564425s) [noise=true , address=45.91.116.14:4191 , piece_id=34U2YIPVQ2WMACJW6JIH7VBI7BDTOSZET4Z6G5VO64O445IQBU2A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.604569054 (2h14m11.644142174s) [noise=true , address=77.90.31.12:4589 , piece_id=TVP3GYPIQS5TF4XRTQXOHTDGKG4RB3W7VBKBYR2WPCIVA24KWGUA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.625211023 (2h14m11.664784153s) [noise=true , address=45.91.116.7:4010 , piece_id=C37EZJFFPR7ZIWBRBPUIDDVQO7GITWTAENVOLYTWAGJXBZJFRKKA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.685780665 (2h14m11.725353785s) [noise=true , address=77.90.31.19:4843 , piece_id=7RBKVWNDFBNI2KG7B4NBQO36ZXI55MF6Q4XWRAX2QSAWRLUKXFJA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.708016543 (2h14m11.747589663s) [noise=true , address=45.91.116.6:4973 , piece_id=GGGR6DT2IPCTLQWCARKUU2PXMA767RN7MTIUW3MEMBTFGJMD3OVA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.784454438 (2h14m11.824027558s) [noise=true , address=77.90.31.14:4661 , piece_id=ERCA4X6WF4TLM4RDSJ2AFVKZSJP77NEIMTJS2PH5V7VAWQDEM7XA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.799625067 (2h14m11.839198187s) [noise=true , address=45.91.116.11:4118 , piece_id=A5BV5Z7AKEBQU3YCROOPRYMBBLONBYFG2TAOO3DI5HINQUGI42SQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.834792140 (2h14m11.87436526s) [noise=true , address=45.91.116.5:4297 , piece_id=5XE77DS5Y3VSYY6WXQEDG4JGCAFGHGQSWSFI5OUD4N2UHJUCG4GA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.896840079 (2h14m11.936413199s) [noise=true , address=45.91.116.12:4225 , piece_id=3NGH2CGXNXICKECZNISSAXSXBIJK4MYAIG3UDIVFGRF5Y5NF7NUQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.908786489 (2h14m11.94835961s) [noise=true , address=77.90.31.16:4733 , piece_id=6ERPNTD7ZGLNV4QISETQBHWN4PIVFTPPUWD4ZHTJYMP5C2FYLE6A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:35.997513589 (2h14m12.037086719s) [noise=true , address=45.91.116.12:4227 , piece_id=7QL7XB4EXJNZLP3G27VN2L3Q6H2YE55NVRJUSY7HE22J7QRZKNIA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472040463 (2h14m12.511613583s) [noise=true , address=77.90.31.16:4734 , piece_id=C5YLG4KBHC5C3P4V4YXQIT7XAN46LFR576RPFLHHDH5YD7NPK5HQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Sep 6 15:42:36.472066212 (2h14m12.511639332s) [seg_pos={0 0}, upload=1948]: Segment reached optimal threshold of 13 pieces. Sep 6 15:42:36.472174839 (2h14m12.51174796s) [noise=true , address=45.91.116.5:4298 , piece_id=TYZBS3TQ5JMUSS764SPL2KRUKK5J52KC42UCNK6N5B5BG4BTYRFA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472215037 (2h14m12.511788157s) [noise=true , address=77.90.31.10:4517 , piece_id=UYHZ3RUG7H3FPD3SIRXQTEHC2MFT6NW6ZHFLOTWOQIKXQC53E5DQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472233703 (2h14m12.511806823s) [noise=true , address=45.91.116.14:4190 , piece_id=BYFKI3FK2EFDELRA6T4SGWBYPQOHEHMA24RYE4FHHN4PLKCGEJGA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Sep 6 15:42:36.472249182 (2h14m12.511822302s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:42:36.472336128 (2h14m12.511909248s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:42:36.472340708 (2h14m12.511913828s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:42:36.839281552 (2h14m12.878854683s) [upload=1948]: Done uploading segments... Sep 6 15:42:36.839305619 (2h14m12.878878789s) [upload=1948]: Upload finished. err: <nil> Duration: 30.003010244s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.214017346s FAIL (2025-09-06 15:42:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.214044778s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.832014371s FAIL (2025-09-06 15:42:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.832037767s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.372581136s FAIL (2025-09-06 15:41:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.372597748s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X.] duration: 12.152166881s FAIL (2025-09-06 15:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:41:05.719643862 (2h12m41.759216982s) [upload=1923]: Starting upload Sep 6 15:41:05.719699278 (2h12m41.759272398s) [upload=1923]: Done starting upload Sep 6 15:41:05.719874343 (2h12m41.759447463s) [upload=1923]: Uploading segments... Sep 6 15:41:05.719906996 (2h12m41.759480116s) [upload=1923]: Got next segment. Inline: false Sep 6 15:41:05.719930782 (2h12m41.759503902s) [upload=1923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:41:05.719935471 (2h12m41.759508591s) [upload=1923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:41:05.970399747 (2h12m42.009972937s) [seg_pos={0 0}, upload=1923]: Begin upload segment... Sep 6 15:41:05.970462638 (2h12m42.010035768s) [seg_pos={0 0}, upload=1923]: Done begin upload segment. Sep 6 15:41:05.970489900 (2h12m42.01006302s) [noise=true , address=77.90.31.18:4807 , piece_id=FJR7WUTFZLYSIM7JRJOKLEHQCM32YV42JLXDE6TUTBTDPKUPN2BQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.970528634 (2h12m42.010101774s) [noise=true , address=45.91.116.7:4011 , piece_id=J62MJXNCKAFEDJMTE4LQP4QQJWDUM45C2YJKW3CHDVQ2CEPHFISQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.970855790 (2h12m42.0104289s) [noise=true , address=45.91.116.9:4261 , piece_id=4DJMZLPYYUDLBUODAJKS3T5DHZPJ4Y56ZNR6CIFFIRYRLEMU6E7A, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971015315 (2h12m42.010588425s) [noise=true , address=45.91.116.13:4155 , piece_id=JR45ZAF6PQP4U6RUBTYCFLGENUKRGQKFCFBW4BBLP3AYGTBWDC6A, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971130285 (2h12m42.010703425s) [noise=true , address=77.90.31.15:4697 , piece_id=GGCR7J6QMLVKGUU7PQDMDSYYVTPKIEGO3B4GOTPGBVNIVMRGPRUA, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971376436 (2h12m42.010949556s) [noise=true , address=77.90.31.18:4805 , piece_id=Q2FGR2GNOMRK7Y2QYR7GMI4OKUJNDSGGZIR4TBCQ4LUWR77EOF7Q, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971716006 (2h12m42.011289126s) [noise=true , address=45.91.116.4:4939 , piece_id=SJNUYHVMUSPHL5GH6S5D3WQI22WCFLVQEK7IRUQAAROXQ4JASDZA, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971759659 (2h12m42.011332769s) [noise=true , address=45.91.116.12:4226 , piece_id=HZZHN3W6HKNTLGOMGSJ4OENCXNEMUQSPSWXLHDJPZ5KEMULW2PUQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971982596 (2h12m42.011555716s) [noise=true , address=45.91.116.5:4299 , piece_id=HXWYWV63HLU35XRTX45M2N3QEJ3WZUS2JSOI6YUUOL4QO5QRPRPQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972013845 (2h12m42.011586965s) [noise=true , address=77.90.31.14:4663 , piece_id=G6EZXY7AZEX72XKU753HJGPRIGKBMJJ4T5OD2AXKNEPM35WMIJ2Q, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972285786 (2h12m42.011858906s) [noise=true , address=45.91.116.9:4263 , piece_id=SQA3DI5ADAG4X2LL4JX5WUTBS2G7FD5EE6BUWSGHZ2JJ5747LQTQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972327466 (2h12m42.011900586s) [noise=true , address=45.91.116.10:4083 , piece_id=7PP33CMLE4UDDWPHWIKVDAZZ3D6GJ7TNQCFJPPNEITMG4P3HXKAQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972518622 (2h12m42.012091742s) [noise=true , address=77.90.31.16:4734 , piece_id=J6V2S5X53PC7LPCU5XPFMVZJJIOGUKT3LPW2P5GBHCJQHBHTNSOQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972646467 (2h12m42.012219617s) [noise=true , address=77.90.31.19:4842 , piece_id=3QUHA35H7OTHAY4FIM2TTG47FQJ27IGBOHVGFMAZAOW6WEF6NFJA, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972771595 (2h12m42.012344716s) [noise=true , address=45.91.116.6:4973 , piece_id=BOR46RBICOCV6GVQS7GI3WTNPMW6WFBX2YDOX73PMQW52WCYMD4A, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.973125413 (2h12m42.012698533s) [noise=true , address=77.90.31.21:4878 , piece_id=XGK5LKSZ5DAKYEHVKCTP4KQKNQOCFXJKDC5BSIGMVGJAXFQGAIWQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:07.816008405 (2h12m43.855581525s) [upload=1923]: Next returned nil segment Sep 6 15:41:07.816024977 (2h12m43.855598097s) [upload=1923]: Waiting for error group managing segments... Sep 6 15:41:08.109327955 (2h12m44.148901075s) [noise=true , address=77.90.31.14:4663 , piece_id=G6EZXY7AZEX72XKU753HJGPRIGKBMJJ4T5OD2AXKNEPM35WMIJ2Q, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.166805119 (2h12m44.206378229s) [noise=true , address=77.90.31.16:4734 , piece_id=J6V2S5X53PC7LPCU5XPFMVZJJIOGUKT3LPW2P5GBHCJQHBHTNSOQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.179492306 (2h12m44.219065426s) [noise=true , address=45.91.116.9:4261 , piece_id=4DJMZLPYYUDLBUODAJKS3T5DHZPJ4Y56ZNR6CIFFIRYRLEMU6E7A, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.200000279 (2h12m44.239573409s) [noise=true , address=45.91.116.5:4299 , piece_id=HXWYWV63HLU35XRTX45M2N3QEJ3WZUS2JSOI6YUUOL4QO5QRPRPQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.207403334 (2h12m44.246976454s) [noise=true , address=77.90.31.19:4842 , piece_id=3QUHA35H7OTHAY4FIM2TTG47FQJ27IGBOHVGFMAZAOW6WEF6NFJA, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.217137190 (2h12m44.25671031s) [noise=true , address=45.91.116.9:4263 , piece_id=SQA3DI5ADAG4X2LL4JX5WUTBS2G7FD5EE6BUWSGHZ2JJ5747LQTQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.246251811 (2h12m44.285824931s) [noise=true , address=45.91.116.6:4973 , piece_id=BOR46RBICOCV6GVQS7GI3WTNPMW6WFBX2YDOX73PMQW52WCYMD4A, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.264490221 (2h12m44.304063331s) [noise=true , address=45.91.116.12:4226 , piece_id=HZZHN3W6HKNTLGOMGSJ4OENCXNEMUQSPSWXLHDJPZ5KEMULW2PUQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.291010207 (2h12m44.330583337s) [noise=true , address=45.91.116.13:4155 , piece_id=JR45ZAF6PQP4U6RUBTYCFLGENUKRGQKFCFBW4BBLP3AYGTBWDC6A, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.512848377 (2h12m44.552421497s) [noise=true , address=77.90.31.18:4807 , piece_id=FJR7WUTFZLYSIM7JRJOKLEHQCM32YV42JLXDE6TUTBTDPKUPN2BQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.522164875 (2h12m44.561737995s) [noise=true , address=45.91.116.10:4083 , piece_id=7PP33CMLE4UDDWPHWIKVDAZZ3D6GJ7TNQCFJPPNEITMG4P3HXKAQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.581761984 (2h12m44.621335094s) [noise=true , address=45.91.116.4:4939 , piece_id=SJNUYHVMUSPHL5GH6S5D3WQI22WCFLVQEK7IRUQAAROXQ4JASDZA, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.811401913 (2h12m44.850975034s) [noise=true , address=77.90.31.21:4878 , piece_id=XGK5LKSZ5DAKYEHVKCTP4KQKNQOCFXJKDC5BSIGMVGJAXFQGAIWQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.811426310 (2h12m44.85099943s) [seg_pos={0 0}, upload=1923]: Segment reached optimal threshold of 13 pieces. Sep 6 15:41:08.811537473 (2h12m44.851110593s) [noise=true , address=77.90.31.15:4697 , piece_id=GGCR7J6QMLVKGUU7PQDMDSYYVTPKIEGO3B4GOTPGBVNIVMRGPRUA, seg_pos={0 0}, upload=1923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 6 15:41:08.811591947 (2h12m44.851165067s) [noise=true , address=77.90.31.18:4805 , piece_id=Q2FGR2GNOMRK7Y2QYR7GMI4OKUJNDSGGZIR4TBCQ4LUWR77EOF7Q, seg_pos={0 0}, upload=1923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Sep 6 15:41:08.811602787 (2h12m44.851175908s) [noise=true , address=45.91.116.7:4011 , piece_id=J62MJXNCKAFEDJMTE4LQP4QQJWDUM45C2YJKW3CHDVQ2CEPHFISQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Sep 6 15:41:08.811615282 (2h12m44.851188402s) [seg_pos={0 0}, upload=1923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:41:08.811698251 (2h12m44.851271371s) [upload=1923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:41:08.811717558 (2h12m44.851290678s) [upload=1923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:41:09.160108528 (2h12m45.199681668s) [upload=1923]: Done uploading segments... Sep 6 15:41:09.160127264 (2h12m45.199700384s) [upload=1923]: Upload finished. err: <nil> Duration: 12.15219257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.X] duration: 30.002067486s FAIL (2025-09-06 15:41:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:41:05.719643862 (2h12m41.759216982s) [upload=1923]: Starting upload Sep 6 15:41:05.719699278 (2h12m41.759272398s) [upload=1923]: Done starting upload Sep 6 15:41:05.719874343 (2h12m41.759447463s) [upload=1923]: Uploading segments... Sep 6 15:41:05.719906996 (2h12m41.759480116s) [upload=1923]: Got next segment. Inline: false Sep 6 15:41:05.719930782 (2h12m41.759503902s) [upload=1923]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:41:05.719935471 (2h12m41.759508591s) [upload=1923]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:41:05.970399747 (2h12m42.009972937s) [seg_pos={0 0}, upload=1923]: Begin upload segment... Sep 6 15:41:05.970462638 (2h12m42.010035768s) [seg_pos={0 0}, upload=1923]: Done begin upload segment. Sep 6 15:41:05.970489900 (2h12m42.01006302s) [noise=true , address=77.90.31.18:4807 , piece_id=FJR7WUTFZLYSIM7JRJOKLEHQCM32YV42JLXDE6TUTBTDPKUPN2BQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.970528634 (2h12m42.010101774s) [noise=true , address=45.91.116.7:4011 , piece_id=J62MJXNCKAFEDJMTE4LQP4QQJWDUM45C2YJKW3CHDVQ2CEPHFISQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.970855790 (2h12m42.0104289s) [noise=true , address=45.91.116.9:4261 , piece_id=4DJMZLPYYUDLBUODAJKS3T5DHZPJ4Y56ZNR6CIFFIRYRLEMU6E7A, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971015315 (2h12m42.010588425s) [noise=true , address=45.91.116.13:4155 , piece_id=JR45ZAF6PQP4U6RUBTYCFLGENUKRGQKFCFBW4BBLP3AYGTBWDC6A, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971130285 (2h12m42.010703425s) [noise=true , address=77.90.31.15:4697 , piece_id=GGCR7J6QMLVKGUU7PQDMDSYYVTPKIEGO3B4GOTPGBVNIVMRGPRUA, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971376436 (2h12m42.010949556s) [noise=true , address=77.90.31.18:4805 , piece_id=Q2FGR2GNOMRK7Y2QYR7GMI4OKUJNDSGGZIR4TBCQ4LUWR77EOF7Q, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971716006 (2h12m42.011289126s) [noise=true , address=45.91.116.4:4939 , piece_id=SJNUYHVMUSPHL5GH6S5D3WQI22WCFLVQEK7IRUQAAROXQ4JASDZA, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971759659 (2h12m42.011332769s) [noise=true , address=45.91.116.12:4226 , piece_id=HZZHN3W6HKNTLGOMGSJ4OENCXNEMUQSPSWXLHDJPZ5KEMULW2PUQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.971982596 (2h12m42.011555716s) [noise=true , address=45.91.116.5:4299 , piece_id=HXWYWV63HLU35XRTX45M2N3QEJ3WZUS2JSOI6YUUOL4QO5QRPRPQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972013845 (2h12m42.011586965s) [noise=true , address=77.90.31.14:4663 , piece_id=G6EZXY7AZEX72XKU753HJGPRIGKBMJJ4T5OD2AXKNEPM35WMIJ2Q, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972285786 (2h12m42.011858906s) [noise=true , address=45.91.116.9:4263 , piece_id=SQA3DI5ADAG4X2LL4JX5WUTBS2G7FD5EE6BUWSGHZ2JJ5747LQTQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972327466 (2h12m42.011900586s) [noise=true , address=45.91.116.10:4083 , piece_id=7PP33CMLE4UDDWPHWIKVDAZZ3D6GJ7TNQCFJPPNEITMG4P3HXKAQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972518622 (2h12m42.012091742s) [noise=true , address=77.90.31.16:4734 , piece_id=J6V2S5X53PC7LPCU5XPFMVZJJIOGUKT3LPW2P5GBHCJQHBHTNSOQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972646467 (2h12m42.012219617s) [noise=true , address=77.90.31.19:4842 , piece_id=3QUHA35H7OTHAY4FIM2TTG47FQJ27IGBOHVGFMAZAOW6WEF6NFJA, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.972771595 (2h12m42.012344716s) [noise=true , address=45.91.116.6:4973 , piece_id=BOR46RBICOCV6GVQS7GI3WTNPMW6WFBX2YDOX73PMQW52WCYMD4A, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:05.973125413 (2h12m42.012698533s) [noise=true , address=77.90.31.21:4878 , piece_id=XGK5LKSZ5DAKYEHVKCTP4KQKNQOCFXJKDC5BSIGMVGJAXFQGAIWQ, seg_pos={0 0}, upload=1923]: Uploading piece... Sep 6 15:41:07.816008405 (2h12m43.855581525s) [upload=1923]: Next returned nil segment Sep 6 15:41:07.816024977 (2h12m43.855598097s) [upload=1923]: Waiting for error group managing segments... Sep 6 15:41:08.109327955 (2h12m44.148901075s) [noise=true , address=77.90.31.14:4663 , piece_id=G6EZXY7AZEX72XKU753HJGPRIGKBMJJ4T5OD2AXKNEPM35WMIJ2Q, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.166805119 (2h12m44.206378229s) [noise=true , address=77.90.31.16:4734 , piece_id=J6V2S5X53PC7LPCU5XPFMVZJJIOGUKT3LPW2P5GBHCJQHBHTNSOQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.179492306 (2h12m44.219065426s) [noise=true , address=45.91.116.9:4261 , piece_id=4DJMZLPYYUDLBUODAJKS3T5DHZPJ4Y56ZNR6CIFFIRYRLEMU6E7A, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.200000279 (2h12m44.239573409s) [noise=true , address=45.91.116.5:4299 , piece_id=HXWYWV63HLU35XRTX45M2N3QEJ3WZUS2JSOI6YUUOL4QO5QRPRPQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.207403334 (2h12m44.246976454s) [noise=true , address=77.90.31.19:4842 , piece_id=3QUHA35H7OTHAY4FIM2TTG47FQJ27IGBOHVGFMAZAOW6WEF6NFJA, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.217137190 (2h12m44.25671031s) [noise=true , address=45.91.116.9:4263 , piece_id=SQA3DI5ADAG4X2LL4JX5WUTBS2G7FD5EE6BUWSGHZ2JJ5747LQTQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.246251811 (2h12m44.285824931s) [noise=true , address=45.91.116.6:4973 , piece_id=BOR46RBICOCV6GVQS7GI3WTNPMW6WFBX2YDOX73PMQW52WCYMD4A, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.264490221 (2h12m44.304063331s) [noise=true , address=45.91.116.12:4226 , piece_id=HZZHN3W6HKNTLGOMGSJ4OENCXNEMUQSPSWXLHDJPZ5KEMULW2PUQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.291010207 (2h12m44.330583337s) [noise=true , address=45.91.116.13:4155 , piece_id=JR45ZAF6PQP4U6RUBTYCFLGENUKRGQKFCFBW4BBLP3AYGTBWDC6A, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.512848377 (2h12m44.552421497s) [noise=true , address=77.90.31.18:4807 , piece_id=FJR7WUTFZLYSIM7JRJOKLEHQCM32YV42JLXDE6TUTBTDPKUPN2BQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.522164875 (2h12m44.561737995s) [noise=true , address=45.91.116.10:4083 , piece_id=7PP33CMLE4UDDWPHWIKVDAZZ3D6GJ7TNQCFJPPNEITMG4P3HXKAQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.581761984 (2h12m44.621335094s) [noise=true , address=45.91.116.4:4939 , piece_id=SJNUYHVMUSPHL5GH6S5D3WQI22WCFLVQEK7IRUQAAROXQ4JASDZA, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.811401913 (2h12m44.850975034s) [noise=true , address=77.90.31.21:4878 , piece_id=XGK5LKSZ5DAKYEHVKCTP4KQKNQOCFXJKDC5BSIGMVGJAXFQGAIWQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: <nil> Sep 6 15:41:08.811426310 (2h12m44.85099943s) [seg_pos={0 0}, upload=1923]: Segment reached optimal threshold of 13 pieces. Sep 6 15:41:08.811537473 (2h12m44.851110593s) [noise=true , address=77.90.31.15:4697 , piece_id=GGCR7J6QMLVKGUU7PQDMDSYYVTPKIEGO3B4GOTPGBVNIVMRGPRUA, seg_pos={0 0}, upload=1923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Sep 6 15:41:08.811591947 (2h12m44.851165067s) [noise=true , address=77.90.31.18:4805 , piece_id=Q2FGR2GNOMRK7Y2QYR7GMI4OKUJNDSGGZIR4TBCQ4LUWR77EOF7Q, seg_pos={0 0}, upload=1923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Sep 6 15:41:08.811602787 (2h12m44.851175908s) [noise=true , address=45.91.116.7:4011 , piece_id=J62MJXNCKAFEDJMTE4LQP4QQJWDUM45C2YJKW3CHDVQ2CEPHFISQ, seg_pos={0 0}, upload=1923]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Sep 6 15:41:08.811615282 (2h12m44.851188402s) [seg_pos={0 0}, upload=1923]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:41:08.811698251 (2h12m44.851271371s) [upload=1923]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:41:08.811717558 (2h12m44.851290678s) [upload=1923]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:41:09.160108528 (2h12m45.199681668s) [upload=1923]: Done uploading segments... Sep 6 15:41:09.160127264 (2h12m45.199700384s) [upload=1923]: Upload finished. err: <nil> Duration: 30.002095338s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.475536518s FAIL (2025-09-06 15:41:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.475551766s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 9.89108213s FAIL (2025-09-06 15:40:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 9.891123248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.326561823s FAIL (2025-09-06 15:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.326581862s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.202972955s FAIL (2025-09-06 15:40:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.20301292s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX.] duration: 7.352173903s FAIL (2025-09-06 15:39:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:38:44.206684817 (2h10m20.246257927s) [upload=1888]: Starting upload Sep 6 15:38:44.206740654 (2h10m20.246313764s) [upload=1888]: Done starting upload Sep 6 15:38:44.206797022 (2h10m20.246370132s) [upload=1888]: Uploading segments... Sep 6 15:38:44.206939744 (2h10m20.246512865s) [upload=1888]: Got next segment. Inline: false Sep 6 15:38:44.206989580 (2h10m20.24656269s) [upload=1888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:38:44.206994339 (2h10m20.246567459s) [upload=1888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:38:44.465068005 (2h10m20.504641125s) [seg_pos={0 0}, upload=1888]: Begin upload segment... Sep 6 15:38:44.465130976 (2h10m20.504704096s) [seg_pos={0 0}, upload=1888]: Done begin upload segment. Sep 6 15:38:44.465156895 (2h10m20.504730045s) [noise=true , address=45.91.116.6:4975 , piece_id=BBRXOOXE36G4CV62LXVXVTPZCMD54FCKJHULGPH442FKNLJ25ZZA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.465170201 (2h10m20.504743311s) [noise=true , address=45.91.116.9:4262 , piece_id=QERRFKZYL4PBTI5EVLBN7OZ6HJJPTSDTK2QPTPPA3M3K4MRIE3WQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.465930206 (2h10m20.505503326s) [noise=true , address=45.91.116.5:4299 , piece_id=CRT4YPGGG3GB2BQYVY5U5S5V2CEFEOKXGCUM5A5QEG2W4B4ISZSQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.466092887 (2h10m20.505666037s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTOVS3KWM63LDYNLFELVCPZRBKAWBYVZE3BEGR7WNWL63XZLCRCQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.467512774 (2h10m20.507085894s) [noise=true , address=45.91.116.14:4191 , piece_id=MAV5VZ7FJKYJLBKEMTZCBTLQ4DQN32IDGBVBRYOEAVBGAYTGFO7A, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.467897731 (2h10m20.507470841s) [noise=true , address=77.90.31.14:4661 , piece_id=BDMWWAWVGEZIVAFZU5ARKBKF7PCIEBF5LB6VKKRXMX3KG6ILVCXQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.468707730 (2h10m20.50828085s) [noise=true , address=77.90.31.17:4771 , piece_id=ADFH4WUPKFWAXNCXVWUZVN52UIILX4O7BXYUBHUQY2A2BJK4VCGA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.469179754 (2h10m20.508752874s) [noise=true , address=45.91.116.13:4153 , piece_id=AADPHS4WPD2CZNROSFTYJKBHAA7JFELV77QZMGT525RXCIRFUXOA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.470389679 (2h10m20.509962799s) [noise=true , address=77.90.31.18:4806 , piece_id=CJJRZRXZD2KKW5CEWI33S43JQK47AGPVL63MI6ZTBS4KUHZYXELQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.471809726 (2h10m20.511382846s) [noise=true , address=45.91.116.5:4297 , piece_id=VOEOOBOBNTZAGOO72KUTELVHRWZA4R6GQCHGODQTZW23OHHPHTGQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.475485529 (2h10m20.515058649s) [noise=true , address=45.91.116.8:4045 , piece_id=PVSFOREEPD566V4KWGNYECQC7MTKXPRWSAYTPL2BNJVAVKOAJNOQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.477839475 (2h10m20.517412595s) [noise=true , address=77.90.31.21:4877 , piece_id=SCVVGOL4S6P2FRVJYOUH6BOHIJW5UGJXYRVYTXFCC4P6W357J3PA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.479488269 (2h10m20.519061389s) [noise=true , address=77.90.31.12:4589 , piece_id=4TMOFJS5RY56TA4CVJ5GJHZ4U3PM54FQWBFNKYD3DM636K4JWWWQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.480685490 (2h10m20.5202586s) [noise=true , address=77.90.31.16:4735 , piece_id=A2NQJSGKYCSTYJZF33GZW5UKHYVRBHMFCNK5EJJO3R2DRID3UK3Q, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.481714609 (2h10m20.521287729s) [noise=true , address=77.90.31.10:4519 , piece_id=YXEPR7Z5XOACJ7N3V6PL5YBEXYUAC2J43SGCCP2QZUU3BQ6HECGQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.482110577 (2h10m20.521683697s) [noise=true , address=77.90.31.10:4518 , piece_id=CRWDRFJSJBFKKPRXZPBCGFJDTW6AU3NC3C3AD6TOMSYJH2C7ASXQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:46.293601760 (2h10m22.3331749s) [upload=1888]: Next returned nil segment Sep 6 15:38:46.293624123 (2h10m22.333197253s) [upload=1888]: Waiting for error group managing segments... Sep 6 15:38:46.590731662 (2h10m22.630304782s) [noise=true , address=45.91.116.14:4191 , piece_id=MAV5VZ7FJKYJLBKEMTZCBTLQ4DQN32IDGBVBRYOEAVBGAYTGFO7A, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.634364654 (2h10m22.673937774s) [noise=true , address=45.91.116.5:4297 , piece_id=VOEOOBOBNTZAGOO72KUTELVHRWZA4R6GQCHGODQTZW23OHHPHTGQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.682877975 (2h10m22.722451096s) [noise=true , address=45.91.116.6:4975 , piece_id=BBRXOOXE36G4CV62LXVXVTPZCMD54FCKJHULGPH442FKNLJ25ZZA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.735607494 (2h10m22.775180614s) [noise=true , address=77.90.31.21:4877 , piece_id=SCVVGOL4S6P2FRVJYOUH6BOHIJW5UGJXYRVYTXFCC4P6W357J3PA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.753122981 (2h10m22.792696101s) [noise=true , address=45.91.116.13:4153 , piece_id=AADPHS4WPD2CZNROSFTYJKBHAA7JFELV77QZMGT525RXCIRFUXOA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.771442214 (2h10m22.811015365s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTOVS3KWM63LDYNLFELVCPZRBKAWBYVZE3BEGR7WNWL63XZLCRCQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.798970300 (2h10m22.83854345s) [noise=true , address=77.90.31.12:4589 , piece_id=4TMOFJS5RY56TA4CVJ5GJHZ4U3PM54FQWBFNKYD3DM636K4JWWWQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.837260819 (2h10m22.876833939s) [noise=true , address=77.90.31.10:4518 , piece_id=CRWDRFJSJBFKKPRXZPBCGFJDTW6AU3NC3C3AD6TOMSYJH2C7ASXQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.957123377 (2h10m22.996696487s) [noise=true , address=77.90.31.10:4519 , piece_id=YXEPR7Z5XOACJ7N3V6PL5YBEXYUAC2J43SGCCP2QZUU3BQ6HECGQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.985365169 (2h10m23.024938289s) [noise=true , address=77.90.31.14:4661 , piece_id=BDMWWAWVGEZIVAFZU5ARKBKF7PCIEBF5LB6VKKRXMX3KG6ILVCXQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.062986969 (2h10m23.102560089s) [noise=true , address=77.90.31.16:4735 , piece_id=A2NQJSGKYCSTYJZF33GZW5UKHYVRBHMFCNK5EJJO3R2DRID3UK3Q, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.078723842 (2h10m23.118296962s) [noise=true , address=77.90.31.18:4806 , piece_id=CJJRZRXZD2KKW5CEWI33S43JQK47AGPVL63MI6ZTBS4KUHZYXELQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.118475096 (2h10m23.158048216s) [noise=true , address=45.91.116.8:4045 , piece_id=PVSFOREEPD566V4KWGNYECQC7MTKXPRWSAYTPL2BNJVAVKOAJNOQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.118502378 (2h10m23.158075498s) [seg_pos={0 0}, upload=1888]: Segment reached optimal threshold of 13 pieces. Sep 6 15:38:47.118620064 (2h10m23.158193184s) [noise=true , address=77.90.31.17:4771 , piece_id=ADFH4WUPKFWAXNCXVWUZVN52UIILX4O7BXYUBHUQY2A2BJK4VCGA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 6 15:38:47.118748209 (2h10m23.158321319s) [noise=true , address=45.91.116.9:4262 , piece_id=QERRFKZYL4PBTI5EVLBN7OZ6HJJPTSDTK2QPTPPA3M3K4MRIE3WQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Sep 6 15:38:47.118822652 (2h10m23.158395772s) [noise=true , address=45.91.116.5:4299 , piece_id=CRT4YPGGG3GB2BQYVY5U5S5V2CEFEOKXGCUM5A5QEG2W4B4ISZSQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:51688->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:51688->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51688->45.91.116.5:4299: use of closed network connection Sep 6 15:38:47.118851547 (2h10m23.158424667s) [seg_pos={0 0}, upload=1888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:38:47.118919437 (2h10m23.158492547s) [upload=1888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:38:47.118923294 (2h10m23.158496414s) [upload=1888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:38:47.432697515 (2h10m23.472270635s) [upload=1888]: Done uploading segments... Sep 6 15:38:47.432716902 (2h10m23.472290022s) [upload=1888]: Upload finished. err: <nil> Duration: 7.352198781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 9.657668546s FAIL (2025-09-06 15:39:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 9.657694536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.980598186s FAIL (2025-09-06 15:39:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.98062054s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X..XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...XX] duration: 30.001595288s FAIL (2025-09-06 15:39:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:38:44.206684817 (2h10m20.246257927s) [upload=1888]: Starting upload Sep 6 15:38:44.206740654 (2h10m20.246313764s) [upload=1888]: Done starting upload Sep 6 15:38:44.206797022 (2h10m20.246370132s) [upload=1888]: Uploading segments... Sep 6 15:38:44.206939744 (2h10m20.246512865s) [upload=1888]: Got next segment. Inline: false Sep 6 15:38:44.206989580 (2h10m20.24656269s) [upload=1888]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:38:44.206994339 (2h10m20.246567459s) [upload=1888]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:38:44.465068005 (2h10m20.504641125s) [seg_pos={0 0}, upload=1888]: Begin upload segment... Sep 6 15:38:44.465130976 (2h10m20.504704096s) [seg_pos={0 0}, upload=1888]: Done begin upload segment. Sep 6 15:38:44.465156895 (2h10m20.504730045s) [noise=true , address=45.91.116.6:4975 , piece_id=BBRXOOXE36G4CV62LXVXVTPZCMD54FCKJHULGPH442FKNLJ25ZZA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.465170201 (2h10m20.504743311s) [noise=true , address=45.91.116.9:4262 , piece_id=QERRFKZYL4PBTI5EVLBN7OZ6HJJPTSDTK2QPTPPA3M3K4MRIE3WQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.465930206 (2h10m20.505503326s) [noise=true , address=45.91.116.5:4299 , piece_id=CRT4YPGGG3GB2BQYVY5U5S5V2CEFEOKXGCUM5A5QEG2W4B4ISZSQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.466092887 (2h10m20.505666037s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTOVS3KWM63LDYNLFELVCPZRBKAWBYVZE3BEGR7WNWL63XZLCRCQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.467512774 (2h10m20.507085894s) [noise=true , address=45.91.116.14:4191 , piece_id=MAV5VZ7FJKYJLBKEMTZCBTLQ4DQN32IDGBVBRYOEAVBGAYTGFO7A, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.467897731 (2h10m20.507470841s) [noise=true , address=77.90.31.14:4661 , piece_id=BDMWWAWVGEZIVAFZU5ARKBKF7PCIEBF5LB6VKKRXMX3KG6ILVCXQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.468707730 (2h10m20.50828085s) [noise=true , address=77.90.31.17:4771 , piece_id=ADFH4WUPKFWAXNCXVWUZVN52UIILX4O7BXYUBHUQY2A2BJK4VCGA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.469179754 (2h10m20.508752874s) [noise=true , address=45.91.116.13:4153 , piece_id=AADPHS4WPD2CZNROSFTYJKBHAA7JFELV77QZMGT525RXCIRFUXOA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.470389679 (2h10m20.509962799s) [noise=true , address=77.90.31.18:4806 , piece_id=CJJRZRXZD2KKW5CEWI33S43JQK47AGPVL63MI6ZTBS4KUHZYXELQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.471809726 (2h10m20.511382846s) [noise=true , address=45.91.116.5:4297 , piece_id=VOEOOBOBNTZAGOO72KUTELVHRWZA4R6GQCHGODQTZW23OHHPHTGQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.475485529 (2h10m20.515058649s) [noise=true , address=45.91.116.8:4045 , piece_id=PVSFOREEPD566V4KWGNYECQC7MTKXPRWSAYTPL2BNJVAVKOAJNOQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.477839475 (2h10m20.517412595s) [noise=true , address=77.90.31.21:4877 , piece_id=SCVVGOL4S6P2FRVJYOUH6BOHIJW5UGJXYRVYTXFCC4P6W357J3PA, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.479488269 (2h10m20.519061389s) [noise=true , address=77.90.31.12:4589 , piece_id=4TMOFJS5RY56TA4CVJ5GJHZ4U3PM54FQWBFNKYD3DM636K4JWWWQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.480685490 (2h10m20.5202586s) [noise=true , address=77.90.31.16:4735 , piece_id=A2NQJSGKYCSTYJZF33GZW5UKHYVRBHMFCNK5EJJO3R2DRID3UK3Q, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.481714609 (2h10m20.521287729s) [noise=true , address=77.90.31.10:4519 , piece_id=YXEPR7Z5XOACJ7N3V6PL5YBEXYUAC2J43SGCCP2QZUU3BQ6HECGQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:44.482110577 (2h10m20.521683697s) [noise=true , address=77.90.31.10:4518 , piece_id=CRWDRFJSJBFKKPRXZPBCGFJDTW6AU3NC3C3AD6TOMSYJH2C7ASXQ, seg_pos={0 0}, upload=1888]: Uploading piece... Sep 6 15:38:46.293601760 (2h10m22.3331749s) [upload=1888]: Next returned nil segment Sep 6 15:38:46.293624123 (2h10m22.333197253s) [upload=1888]: Waiting for error group managing segments... Sep 6 15:38:46.590731662 (2h10m22.630304782s) [noise=true , address=45.91.116.14:4191 , piece_id=MAV5VZ7FJKYJLBKEMTZCBTLQ4DQN32IDGBVBRYOEAVBGAYTGFO7A, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.634364654 (2h10m22.673937774s) [noise=true , address=45.91.116.5:4297 , piece_id=VOEOOBOBNTZAGOO72KUTELVHRWZA4R6GQCHGODQTZW23OHHPHTGQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.682877975 (2h10m22.722451096s) [noise=true , address=45.91.116.6:4975 , piece_id=BBRXOOXE36G4CV62LXVXVTPZCMD54FCKJHULGPH442FKNLJ25ZZA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.735607494 (2h10m22.775180614s) [noise=true , address=77.90.31.21:4877 , piece_id=SCVVGOL4S6P2FRVJYOUH6BOHIJW5UGJXYRVYTXFCC4P6W357J3PA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.753122981 (2h10m22.792696101s) [noise=true , address=45.91.116.13:4153 , piece_id=AADPHS4WPD2CZNROSFTYJKBHAA7JFELV77QZMGT525RXCIRFUXOA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.771442214 (2h10m22.811015365s) [noise=true , address=45.91.116.10:4083 , piece_id=ZTOVS3KWM63LDYNLFELVCPZRBKAWBYVZE3BEGR7WNWL63XZLCRCQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.798970300 (2h10m22.83854345s) [noise=true , address=77.90.31.12:4589 , piece_id=4TMOFJS5RY56TA4CVJ5GJHZ4U3PM54FQWBFNKYD3DM636K4JWWWQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.837260819 (2h10m22.876833939s) [noise=true , address=77.90.31.10:4518 , piece_id=CRWDRFJSJBFKKPRXZPBCGFJDTW6AU3NC3C3AD6TOMSYJH2C7ASXQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.957123377 (2h10m22.996696487s) [noise=true , address=77.90.31.10:4519 , piece_id=YXEPR7Z5XOACJ7N3V6PL5YBEXYUAC2J43SGCCP2QZUU3BQ6HECGQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:46.985365169 (2h10m23.024938289s) [noise=true , address=77.90.31.14:4661 , piece_id=BDMWWAWVGEZIVAFZU5ARKBKF7PCIEBF5LB6VKKRXMX3KG6ILVCXQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.062986969 (2h10m23.102560089s) [noise=true , address=77.90.31.16:4735 , piece_id=A2NQJSGKYCSTYJZF33GZW5UKHYVRBHMFCNK5EJJO3R2DRID3UK3Q, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.078723842 (2h10m23.118296962s) [noise=true , address=77.90.31.18:4806 , piece_id=CJJRZRXZD2KKW5CEWI33S43JQK47AGPVL63MI6ZTBS4KUHZYXELQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.118475096 (2h10m23.158048216s) [noise=true , address=45.91.116.8:4045 , piece_id=PVSFOREEPD566V4KWGNYECQC7MTKXPRWSAYTPL2BNJVAVKOAJNOQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: <nil> Sep 6 15:38:47.118502378 (2h10m23.158075498s) [seg_pos={0 0}, upload=1888]: Segment reached optimal threshold of 13 pieces. Sep 6 15:38:47.118620064 (2h10m23.158193184s) [noise=true , address=77.90.31.17:4771 , piece_id=ADFH4WUPKFWAXNCXVWUZVN52UIILX4O7BXYUBHUQY2A2BJK4VCGA, seg_pos={0 0}, upload=1888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: expected piece hash; context canceled; EOF Sep 6 15:38:47.118748209 (2h10m23.158321319s) [noise=true , address=45.91.116.9:4262 , piece_id=QERRFKZYL4PBTI5EVLBN7OZ6HJJPTSDTK2QPTPPA3M3K4MRIE3WQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Sep 6 15:38:47.118822652 (2h10m23.158395772s) [noise=true , address=45.91.116.5:4299 , piece_id=CRT4YPGGG3GB2BQYVY5U5S5V2CEFEOKXGCUM5A5QEG2W4B4ISZSQ, seg_pos={0 0}, upload=1888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: write tcp 188.166.225.170:51688->45.91.116.5:4299: use of closed network connection; write tcp 188.166.225.170:51688->45.91.116.5:4299: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51688->45.91.116.5:4299: use of closed network connection Sep 6 15:38:47.118851547 (2h10m23.158424667s) [seg_pos={0 0}, upload=1888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:38:47.118919437 (2h10m23.158492547s) [upload=1888]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:38:47.118923294 (2h10m23.158496414s) [upload=1888]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:38:47.432697515 (2h10m23.472270635s) [upload=1888]: Done uploading segments... Sep 6 15:38:47.432716902 (2h10m23.472290022s) [upload=1888]: Upload finished. err: <nil> Duration: 30.001616548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 37.447769237s FAIL (2025-09-06 15:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 37.447795427s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.989965211s FAIL (2025-09-06 15:38:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.989987242s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X..XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...X] duration: 30.002295214s FAIL (2025-09-06 15:38:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:37:51.237190684 (2h9m27.276763804s) [upload=1874]: Starting upload Sep 6 15:37:51.237254245 (2h9m27.276827365s) [upload=1874]: Done starting upload Sep 6 15:37:51.237361641 (2h9m27.276934761s) [upload=1874]: Uploading segments... Sep 6 15:37:51.237445591 (2h9m27.277018711s) [upload=1874]: Got next segment. Inline: false Sep 6 15:37:51.237479026 (2h9m27.277052136s) [upload=1874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:37:51.237486270 (2h9m27.27705938s) [upload=1874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:37:51.559755787 (2h9m27.599328917s) [seg_pos={0 0}, upload=1874]: Begin upload segment... Sep 6 15:37:51.559797326 (2h9m27.599370446s) [seg_pos={0 0}, upload=1874]: Done begin upload segment. Sep 6 15:37:51.559822685 (2h9m27.599395805s) [noise=true , address=77.90.31.18:4805 , piece_id=5UDQ62PPTW2CBKYCKJEB6V3WAB44MH36CMLVFNO6BN7QXUSS7Q5Q, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.559874815 (2h9m27.599447935s) [noise=true , address=45.91.116.6:4973 , piece_id=CRNQJ2OPKBLN4OSAXTCHXV6R3DXMYNWCX62Y7NTNXUDASZ57VCHA, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560186471 (2h9m27.599759591s) [noise=true , address=77.90.31.21:4878 , piece_id=GV2OQRSVPUPLCAQR5O5KBQAVJOXY2AOUZ37WV4P5IQNXQIUJWIGQ, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560287454 (2h9m27.599860574s) [noise=true , address=45.91.116.7:4011 , piece_id=RRB36WCXTZY7Y4OGY7OQKSGEV5ZRRJULNOJJVPFGA5XTZ3EDD35A, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560444876 (2h9m27.600017996s) [noise=true , address=45.91.116.8:4048 , piece_id=XD4VSZO4AMRRNSNPWA4NIKXVGREXJ3TVEXDTVPY4APO225EEYQVQ, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560544506 (2h9m27.600117626s) [noise=true , address=77.90.31.19:4841 , piece_id=NXMMTZBD2SYZ2SYYXKYELIGYKCU4N4AGFU6CNGQILCR644VWHHBQ, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560702839 (2h9m27.600275959s) [noise=true , address=77.90.31.10:4519 , piece_id=U52KSIBCKDR5VT5MOXTNEBK42X6MIPBJL7RZOR4JDIMY2XPBIRJA, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560810024 (2h9m27.600383144s) [noise=true , address=45.91.116.4:4939 , piece_id=E7HR5VVG6TNK73LPQWFL3AIWKFGO3A6TOIRWNHCROHPWZOZZSFBA, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.560948228 (2h9m27.600521348s) [noise=true , address=77.90.31.14:4662 , piece_id=DGVO2KWQM643FX6D2MPQLITDHJJCFYMNGFYRJSEZKEH22NYVVVCA, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561059301 (2h9m27.600632421s) [noise=true , address=77.90.31.21:4879 , piece_id=ECV5CKW5P7XZEOCUX4JMKKXE5B6K52B7Q5URXWSIM7UZVCFTVWHQ, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561167929 (2h9m27.600741049s) [noise=true , address=45.91.116.11:4117 , piece_id=35PDMQUAPLRGVRVXSUKMUKUIWTHTTDM45WFKQOWNKRBQ3WBCJMTA, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561300953 (2h9m27.600874093s) [noise=true , address=45.91.116.7:4010 , piece_id=YE3ETTLZQ7V2PNHIGZJEVJHEF2QFITOIDLVPD2GXBEQHD7HMH3CA, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561384383 (2h9m27.600957503s) [noise=true , address=77.90.31.13:4626 , piece_id=A6KYD25XQDZ4D4STAHZ6QJYSTPMOLAM3PAFMQC3INZ7STQ7BNQ6Q, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561535251 (2h9m27.601108371s) [noise=true , address=45.91.116.10:4083 , piece_id=D5PDPJQB2HJOKLSLE5C7R7DKNUYHV6BZH6WBWBDUPKPJZANE5M4A, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561603492 (2h9m27.601176612s) [noise=true , address=45.91.116.9:4262 , piece_id=SCXFAOUVNIQ5FZUTUAZOVGEGLFPF44RYU2LW5VGDVC74N4H57I6A, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:51.561866696 (2h9m27.601439816s) [noise=true , address=77.90.31.10:4517 , piece_id=TYDZQ4G2NSNIIHV7ZBZTB6IHAJ4TC2OG3EITQX4YHIHZ4WCDCK4Q, seg_pos={0 0}, upload=1874]: Uploading piece... Sep 6 15:37:53.293501673 (2h9m29.333074803s) [upload=1874]: Next returned nil segment Sep 6 15:37:53.293516522 (2h9m29.333089642s) [upload=1874]: Waiting for error group managing segments... Sep 6 15:37:53.704206542 (2h9m29.743779662s) [noise=true , address=45.91.116.9:4262 , piece_id=SCXFAOUVNIQ5FZUTUAZOVGEGLFPF44RYU2LW5VGDVC74N4H57I6A, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.745783620 (2h9m29.78535674s) [noise=true , address=45.91.116.11:4117 , piece_id=35PDMQUAPLRGVRVXSUKMUKUIWTHTTDM45WFKQOWNKRBQ3WBCJMTA, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.775338575 (2h9m29.814911695s) [noise=true , address=45.91.116.7:4010 , piece_id=YE3ETTLZQ7V2PNHIGZJEVJHEF2QFITOIDLVPD2GXBEQHD7HMH3CA, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.794516672 (2h9m29.834089792s) [noise=true , address=77.90.31.14:4662 , piece_id=DGVO2KWQM643FX6D2MPQLITDHJJCFYMNGFYRJSEZKEH22NYVVVCA, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.804824386 (2h9m29.844397506s) [noise=true , address=77.90.31.19:4841 , piece_id=NXMMTZBD2SYZ2SYYXKYELIGYKCU4N4AGFU6CNGQILCR644VWHHBQ, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.819326096 (2h9m29.858899216s) [noise=true , address=45.91.116.6:4973 , piece_id=CRNQJ2OPKBLN4OSAXTCHXV6R3DXMYNWCX62Y7NTNXUDASZ57VCHA, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.891263799 (2h9m29.930836909s) [noise=true , address=77.90.31.10:4517 , piece_id=TYDZQ4G2NSNIIHV7ZBZTB6IHAJ4TC2OG3EITQX4YHIHZ4WCDCK4Q, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.924466543 (2h9m29.964039663s) [noise=true , address=45.91.116.8:4048 , piece_id=XD4VSZO4AMRRNSNPWA4NIKXVGREXJ3TVEXDTVPY4APO225EEYQVQ, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:53.961898940 (2h9m30.00147206s) [noise=true , address=77.90.31.18:4805 , piece_id=5UDQ62PPTW2CBKYCKJEB6V3WAB44MH36CMLVFNO6BN7QXUSS7Q5Q, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:54.024077680 (2h9m30.0636508s) [noise=true , address=45.91.116.10:4083 , piece_id=D5PDPJQB2HJOKLSLE5C7R7DKNUYHV6BZH6WBWBDUPKPJZANE5M4A, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:54.069792926 (2h9m30.109366047s) [noise=true , address=77.90.31.21:4878 , piece_id=GV2OQRSVPUPLCAQR5O5KBQAVJOXY2AOUZ37WV4P5IQNXQIUJWIGQ, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:54.206479779 (2h9m30.246052909s) [noise=true , address=77.90.31.21:4879 , piece_id=ECV5CKW5P7XZEOCUX4JMKKXE5B6K52B7Q5URXWSIM7UZVCFTVWHQ, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:54.382642609 (2h9m30.422215719s) [noise=true , address=45.91.116.4:4939 , piece_id=E7HR5VVG6TNK73LPQWFL3AIWKFGO3A6TOIRWNHCROHPWZOZZSFBA, seg_pos={0 0}, upload=1874]: Done uploading piece. err: <nil> Sep 6 15:37:54.382670542 (2h9m30.422243662s) [seg_pos={0 0}, upload=1874]: Segment reached optimal threshold of 13 pieces. Sep 6 15:37:54.382827903 (2h9m30.422401024s) [noise=true , address=77.90.31.10:4519 , piece_id=U52KSIBCKDR5VT5MOXTNEBK42X6MIPBJL7RZOR4JDIMY2XPBIRJA, seg_pos={0 0}, upload=1874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Sep 6 15:37:54.382910131 (2h9m30.422483251s) [noise=true , address=77.90.31.13:4626 , piece_id=A6KYD25XQDZ4D4STAHZ6QJYSTPMOLAM3PAFMQC3INZ7STQ7BNQ6Q, seg_pos={0 0}, upload=1874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 6 15:37:54.383016114 (2h9m30.422589234s) [noise=true , address=45.91.116.7:4011 , piece_id=RRB36WCXTZY7Y4OGY7OQKSGEV5ZRRJULNOJJVPFGA5XTZ3EDD35A, seg_pos={0 0}, upload=1874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: read tcp 188.166.225.170:48870->45.91.116.7:4011: use of closed network connection; context canceled Sep 6 15:37:54.383047744 (2h9m30.422620864s) [seg_pos={0 0}, upload=1874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:37:54.383114472 (2h9m30.422687592s) [upload=1874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:37:54.383118550 (2h9m30.42269167s) [upload=1874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:37:54.897934767 (2h9m30.937507878s) [upload=1874]: Done uploading segments... Sep 6 15:37:54.897951961 (2h9m30.937525081s) [upload=1874]: Upload finished. err: <nil> Duration: 30.002316454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 14.301594628s FAIL (2025-09-06 15:38:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 14.301622842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.041132038s FAIL (2025-09-06 15:37:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.04120115s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 13.871474456s FAIL (2025-09-06 15:37:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 13.871493842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...X..XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X...] duration: 21.798616588s FAIL (2025-09-06 15:37:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:34:06.526298371 (2h5m42.565871491s) [upload=1821]: Starting upload Sep 6 15:34:06.526369577 (2h5m42.565942697s) [upload=1821]: Done starting upload Sep 6 15:34:06.526472745 (2h5m42.566045865s) [upload=1821]: Uploading segments... Sep 6 15:34:06.526567155 (2h5m42.566140275s) [upload=1821]: Got next segment. Inline: false Sep 6 15:34:06.526599297 (2h5m42.566172407s) [upload=1821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:34:06.526603515 (2h5m42.566176635s) [upload=1821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:34:06.773332114 (2h5m42.812905234s) [seg_pos={0 0}, upload=1821]: Begin upload segment... Sep 6 15:34:06.773384955 (2h5m42.812958075s) [seg_pos={0 0}, upload=1821]: Done begin upload segment. Sep 6 15:34:06.773406065 (2h5m42.812979185s) [noise=true , address=45.91.116.10:4082 , piece_id=NDW5NGE6DUDVHVFSEWKS3ARHQ3G6PQSJRZWFMEU24SH36BLENMHQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.773427175 (2h5m42.813000295s) [noise=true , address=77.90.31.14:4663 , piece_id=WFOXSLQOBNF7KZSQEG6ZBVZCENKLZKTJQPVJOVIU4ZYFZENTREDA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.773774961 (2h5m42.813348081s) [noise=true , address=45.91.116.6:4973 , piece_id=RARJ2DNHC3YMBJH37WI7PRVV4X2AN2Q3UW5GBWJTB3ZVK42ZUCRQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.773900912 (2h5m42.813474032s) [noise=true , address=45.91.116.7:4009 , piece_id=3TH3ZFJUVXMC6MRF2OGM4BEJUAAJT3BAXMXSQE7UNG4XMAF32CDQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774075716 (2h5m42.813648836s) [noise=true , address=45.91.116.12:4226 , piece_id=HTZEDHVB5TAG4G3B4ZMDUZD5MNSWDXTGMISCDAPT75EG26HHBFNQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774329121 (2h5m42.813902241s) [noise=true , address=45.91.116.12:4227 , piece_id=SYVYKRAC4AYVEG7RYXCJ6AVIJDL3WT7KH7BVW27M4HD4ITJLG54A, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774456475 (2h5m42.814029595s) [noise=true , address=45.91.116.10:4083 , piece_id=YUSPMDWLP3XQXBJJLDGT3ZIQEAE2QCNUTHDQRCEXH3YNIIPL2PDA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774640537 (2h5m42.814213657s) [noise=true , address=45.91.116.13:4153 , piece_id=OESJ4O33LICXWSUEKCR7R23FBFO4PRMAODLCOSQZXIJL67ARQGPQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774916926 (2h5m42.814490046s) [noise=true , address=77.90.31.19:4842 , piece_id=TCOC32BPUJNCYWFA65JELIDUFVXQLZT26F56JEKNKUQK2TPZW2RA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774937436 (2h5m42.814510556s) [noise=true , address=77.90.31.18:4805 , piece_id=Q64OLFZHCR4MXBX7YNXMWXYEXI4ITJ63PWTXCG4E4AADXKIIECNQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775281022 (2h5m42.814854143s) [noise=true , address=45.91.116.14:4191 , piece_id=KQSDL4OTXSEKRNYMCTRFND6X2NRMS2QWAPDHNY5AP5YVUAKMFLPQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775333964 (2h5m42.814907084s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5AQNPMEIEZK4JZ6RX5SDAWE34HJSREERTLBCTOBEXI6HIHLTEQA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775626093 (2h5m42.815199213s) [noise=true , address=77.90.31.12:4592 , piece_id=Z6GDY5S3EONXUP2AKMHREZHFFQAM45WCE6BZVV3OAPAEMQGUETBA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775707709 (2h5m42.815280829s) [noise=true , address=45.91.116.6:4974 , piece_id=DYTZZ2CNSDR4VK2XKWNXWQRQU75E3KXE2QO7CMR2J5WZGDBWRHRA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.776086935 (2h5m42.815660055s) [noise=true , address=77.90.31.12:4589 , piece_id=HVRO4TQGGU5LDSUBUURQDK5WUUTESKDFB2XALUEQBM6LNLLUMYHA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.776312456 (2h5m42.815885576s) [noise=true , address=77.90.31.10:4519 , piece_id=H6A6I5EMXLMQIVT32UKOTZSXVV4T3D34CPVPLYQSEWGRFQ3MK27Q, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:08.629698566 (2h5m44.669271686s) [upload=1821]: Next returned nil segment Sep 6 15:34:08.629719456 (2h5m44.669292576s) [upload=1821]: Waiting for error group managing segments... Sep 6 15:34:08.924034281 (2h5m44.963607401s) [noise=true , address=45.91.116.13:4153 , piece_id=OESJ4O33LICXWSUEKCR7R23FBFO4PRMAODLCOSQZXIJL67ARQGPQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:08.958624741 (2h5m44.998197861s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5AQNPMEIEZK4JZ6RX5SDAWE34HJSREERTLBCTOBEXI6HIHLTEQA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:08.961913433 (2h5m45.001486554s) [noise=true , address=77.90.31.10:4519 , piece_id=H6A6I5EMXLMQIVT32UKOTZSXVV4T3D34CPVPLYQSEWGRFQ3MK27Q, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:08.975852506 (2h5m45.015425626s) [noise=true , address=45.91.116.14:4191 , piece_id=KQSDL4OTXSEKRNYMCTRFND6X2NRMS2QWAPDHNY5AP5YVUAKMFLPQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.051854647 (2h5m45.091427767s) [noise=true , address=45.91.116.12:4227 , piece_id=SYVYKRAC4AYVEG7RYXCJ6AVIJDL3WT7KH7BVW27M4HD4ITJLG54A, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.073992769 (2h5m45.113565889s) [noise=true , address=77.90.31.12:4592 , piece_id=Z6GDY5S3EONXUP2AKMHREZHFFQAM45WCE6BZVV3OAPAEMQGUETBA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.092927441 (2h5m45.132500561s) [noise=true , address=45.91.116.6:4973 , piece_id=RARJ2DNHC3YMBJH37WI7PRVV4X2AN2Q3UW5GBWJTB3ZVK42ZUCRQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.119079864 (2h5m45.158652984s) [noise=true , address=77.90.31.14:4663 , piece_id=WFOXSLQOBNF7KZSQEG6ZBVZCENKLZKTJQPVJOVIU4ZYFZENTREDA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.152489253 (2h5m45.192062373s) [noise=true , address=45.91.116.7:4009 , piece_id=3TH3ZFJUVXMC6MRF2OGM4BEJUAAJT3BAXMXSQE7UNG4XMAF32CDQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.158673917 (2h5m45.198247037s) [noise=true , address=45.91.116.12:4226 , piece_id=HTZEDHVB5TAG4G3B4ZMDUZD5MNSWDXTGMISCDAPT75EG26HHBFNQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.189132660 (2h5m45.22870578s) [noise=true , address=77.90.31.19:4842 , piece_id=TCOC32BPUJNCYWFA65JELIDUFVXQLZT26F56JEKNKUQK2TPZW2RA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.252200842 (2h5m45.291773972s) [noise=true , address=77.90.31.18:4805 , piece_id=Q64OLFZHCR4MXBX7YNXMWXYEXI4ITJ63PWTXCG4E4AADXKIIECNQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.306187597 (2h5m45.345760717s) [noise=true , address=77.90.31.12:4589 , piece_id=HVRO4TQGGU5LDSUBUURQDK5WUUTESKDFB2XALUEQBM6LNLLUMYHA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.306222354 (2h5m45.345795474s) [seg_pos={0 0}, upload=1821]: Segment reached optimal threshold of 13 pieces. Sep 6 15:34:09.306455760 (2h5m45.34602888s) [noise=true , address=45.91.116.10:4082 , piece_id=NDW5NGE6DUDVHVFSEWKS3ARHQ3G6PQSJRZWFMEU24SH36BLENMHQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 6 15:34:09.306563526 (2h5m45.346136646s) [noise=true , address=45.91.116.6:4974 , piece_id=DYTZZ2CNSDR4VK2XKWNXWQRQU75E3KXE2QO7CMR2J5WZGDBWRHRA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 6 15:34:09.306621858 (2h5m45.346195049s) [noise=true , address=45.91.116.10:4083 , piece_id=YUSPMDWLP3XQXBJJLDGT3ZIQEAE2QCNUTHDQRCEXH3YNIIPL2PDA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35772->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35772->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35772->45.91.116.10:4083: use of closed network connection Sep 6 15:34:09.306671633 (2h5m45.346244753s) [seg_pos={0 0}, upload=1821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:34:09.306787064 (2h5m45.346360185s) [upload=1821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:34:09.306791713 (2h5m45.346364833s) [upload=1821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:34:09.676642167 (2h5m45.716215287s) [upload=1821]: Done uploading segments... Sep 6 15:34:09.676661113 (2h5m45.716234234s) [upload=1821]: Upload finished. err: <nil> Duration: 21.798642076s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 17.811081625s FAIL (2025-09-06 15:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.811102666s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 18.835421264s FAIL (2025-09-06 15:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 18.835437745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001382743s FAIL (2025-09-06 15:36:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001402041s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 32.739151162s FAIL (2025-09-06 15:36:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 32.739177603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X...X..XX...X.X...XX.XXX..XX..X.XX..XXX...X.XXX..XX..XX..X..] duration: 8.522202932s FAIL (2025-09-06 15:36:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 15:34:06.526298371 (2h5m42.565871491s) [upload=1821]: Starting upload Sep 6 15:34:06.526369577 (2h5m42.565942697s) [upload=1821]: Done starting upload Sep 6 15:34:06.526472745 (2h5m42.566045865s) [upload=1821]: Uploading segments... Sep 6 15:34:06.526567155 (2h5m42.566140275s) [upload=1821]: Got next segment. Inline: false Sep 6 15:34:06.526599297 (2h5m42.566172407s) [upload=1821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 15:34:06.526603515 (2h5m42.566176635s) [upload=1821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 15:34:06.773332114 (2h5m42.812905234s) [seg_pos={0 0}, upload=1821]: Begin upload segment... Sep 6 15:34:06.773384955 (2h5m42.812958075s) [seg_pos={0 0}, upload=1821]: Done begin upload segment. Sep 6 15:34:06.773406065 (2h5m42.812979185s) [noise=true , address=45.91.116.10:4082 , piece_id=NDW5NGE6DUDVHVFSEWKS3ARHQ3G6PQSJRZWFMEU24SH36BLENMHQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.773427175 (2h5m42.813000295s) [noise=true , address=77.90.31.14:4663 , piece_id=WFOXSLQOBNF7KZSQEG6ZBVZCENKLZKTJQPVJOVIU4ZYFZENTREDA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.773774961 (2h5m42.813348081s) [noise=true , address=45.91.116.6:4973 , piece_id=RARJ2DNHC3YMBJH37WI7PRVV4X2AN2Q3UW5GBWJTB3ZVK42ZUCRQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.773900912 (2h5m42.813474032s) [noise=true , address=45.91.116.7:4009 , piece_id=3TH3ZFJUVXMC6MRF2OGM4BEJUAAJT3BAXMXSQE7UNG4XMAF32CDQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774075716 (2h5m42.813648836s) [noise=true , address=45.91.116.12:4226 , piece_id=HTZEDHVB5TAG4G3B4ZMDUZD5MNSWDXTGMISCDAPT75EG26HHBFNQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774329121 (2h5m42.813902241s) [noise=true , address=45.91.116.12:4227 , piece_id=SYVYKRAC4AYVEG7RYXCJ6AVIJDL3WT7KH7BVW27M4HD4ITJLG54A, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774456475 (2h5m42.814029595s) [noise=true , address=45.91.116.10:4083 , piece_id=YUSPMDWLP3XQXBJJLDGT3ZIQEAE2QCNUTHDQRCEXH3YNIIPL2PDA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774640537 (2h5m42.814213657s) [noise=true , address=45.91.116.13:4153 , piece_id=OESJ4O33LICXWSUEKCR7R23FBFO4PRMAODLCOSQZXIJL67ARQGPQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774916926 (2h5m42.814490046s) [noise=true , address=77.90.31.19:4842 , piece_id=TCOC32BPUJNCYWFA65JELIDUFVXQLZT26F56JEKNKUQK2TPZW2RA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.774937436 (2h5m42.814510556s) [noise=true , address=77.90.31.18:4805 , piece_id=Q64OLFZHCR4MXBX7YNXMWXYEXI4ITJ63PWTXCG4E4AADXKIIECNQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775281022 (2h5m42.814854143s) [noise=true , address=45.91.116.14:4191 , piece_id=KQSDL4OTXSEKRNYMCTRFND6X2NRMS2QWAPDHNY5AP5YVUAKMFLPQ, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775333964 (2h5m42.814907084s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5AQNPMEIEZK4JZ6RX5SDAWE34HJSREERTLBCTOBEXI6HIHLTEQA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775626093 (2h5m42.815199213s) [noise=true , address=77.90.31.12:4592 , piece_id=Z6GDY5S3EONXUP2AKMHREZHFFQAM45WCE6BZVV3OAPAEMQGUETBA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.775707709 (2h5m42.815280829s) [noise=true , address=45.91.116.6:4974 , piece_id=DYTZZ2CNSDR4VK2XKWNXWQRQU75E3KXE2QO7CMR2J5WZGDBWRHRA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.776086935 (2h5m42.815660055s) [noise=true , address=77.90.31.12:4589 , piece_id=HVRO4TQGGU5LDSUBUURQDK5WUUTESKDFB2XALUEQBM6LNLLUMYHA, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:06.776312456 (2h5m42.815885576s) [noise=true , address=77.90.31.10:4519 , piece_id=H6A6I5EMXLMQIVT32UKOTZSXVV4T3D34CPVPLYQSEWGRFQ3MK27Q, seg_pos={0 0}, upload=1821]: Uploading piece... Sep 6 15:34:08.629698566 (2h5m44.669271686s) [upload=1821]: Next returned nil segment Sep 6 15:34:08.629719456 (2h5m44.669292576s) [upload=1821]: Waiting for error group managing segments... Sep 6 15:34:08.924034281 (2h5m44.963607401s) [noise=true , address=45.91.116.13:4153 , piece_id=OESJ4O33LICXWSUEKCR7R23FBFO4PRMAODLCOSQZXIJL67ARQGPQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:08.958624741 (2h5m44.998197861s) [noise=true , address=45.91.116.4:4938 , piece_id=Q5AQNPMEIEZK4JZ6RX5SDAWE34HJSREERTLBCTOBEXI6HIHLTEQA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:08.961913433 (2h5m45.001486554s) [noise=true , address=77.90.31.10:4519 , piece_id=H6A6I5EMXLMQIVT32UKOTZSXVV4T3D34CPVPLYQSEWGRFQ3MK27Q, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:08.975852506 (2h5m45.015425626s) [noise=true , address=45.91.116.14:4191 , piece_id=KQSDL4OTXSEKRNYMCTRFND6X2NRMS2QWAPDHNY5AP5YVUAKMFLPQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.051854647 (2h5m45.091427767s) [noise=true , address=45.91.116.12:4227 , piece_id=SYVYKRAC4AYVEG7RYXCJ6AVIJDL3WT7KH7BVW27M4HD4ITJLG54A, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.073992769 (2h5m45.113565889s) [noise=true , address=77.90.31.12:4592 , piece_id=Z6GDY5S3EONXUP2AKMHREZHFFQAM45WCE6BZVV3OAPAEMQGUETBA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.092927441 (2h5m45.132500561s) [noise=true , address=45.91.116.6:4973 , piece_id=RARJ2DNHC3YMBJH37WI7PRVV4X2AN2Q3UW5GBWJTB3ZVK42ZUCRQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.119079864 (2h5m45.158652984s) [noise=true , address=77.90.31.14:4663 , piece_id=WFOXSLQOBNF7KZSQEG6ZBVZCENKLZKTJQPVJOVIU4ZYFZENTREDA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.152489253 (2h5m45.192062373s) [noise=true , address=45.91.116.7:4009 , piece_id=3TH3ZFJUVXMC6MRF2OGM4BEJUAAJT3BAXMXSQE7UNG4XMAF32CDQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.158673917 (2h5m45.198247037s) [noise=true , address=45.91.116.12:4226 , piece_id=HTZEDHVB5TAG4G3B4ZMDUZD5MNSWDXTGMISCDAPT75EG26HHBFNQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.189132660 (2h5m45.22870578s) [noise=true , address=77.90.31.19:4842 , piece_id=TCOC32BPUJNCYWFA65JELIDUFVXQLZT26F56JEKNKUQK2TPZW2RA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.252200842 (2h5m45.291773972s) [noise=true , address=77.90.31.18:4805 , piece_id=Q64OLFZHCR4MXBX7YNXMWXYEXI4ITJ63PWTXCG4E4AADXKIIECNQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.306187597 (2h5m45.345760717s) [noise=true , address=77.90.31.12:4589 , piece_id=HVRO4TQGGU5LDSUBUURQDK5WUUTESKDFB2XALUEQBM6LNLLUMYHA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: <nil> Sep 6 15:34:09.306222354 (2h5m45.345795474s) [seg_pos={0 0}, upload=1821]: Segment reached optimal threshold of 13 pieces. Sep 6 15:34:09.306455760 (2h5m45.34602888s) [noise=true , address=45.91.116.10:4082 , piece_id=NDW5NGE6DUDVHVFSEWKS3ARHQ3G6PQSJRZWFMEU24SH36BLENMHQ, seg_pos={0 0}, upload=1821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Sep 6 15:34:09.306563526 (2h5m45.346136646s) [noise=true , address=45.91.116.6:4974 , piece_id=DYTZZ2CNSDR4VK2XKWNXWQRQU75E3KXE2QO7CMR2J5WZGDBWRHRA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Sep 6 15:34:09.306621858 (2h5m45.346195049s) [noise=true , address=45.91.116.10:4083 , piece_id=YUSPMDWLP3XQXBJJLDGT3ZIQEAE2QCNUTHDQRCEXH3YNIIPL2PDA, seg_pos={0 0}, upload=1821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:35772->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:35772->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35772->45.91.116.10:4083: use of closed network connection Sep 6 15:34:09.306671633 (2h5m45.346244753s) [seg_pos={0 0}, upload=1821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 6 15:34:09.306787064 (2h5m45.346360185s) [upload=1821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 15:34:09.306791713 (2h5m45.346364833s) [upload=1821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 15:34:09.676642167 (2h5m45.716215287s) [upload=1821]: Done uploading segments... Sep 6 15:34:09.676661113 (2h5m45.716234234s) [upload=1821]: Upload finished. err: <nil> Duration: 8.522225855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 12.422650732s FAIL (2025-09-06 15:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Sep 6 14:53:56.468793006 (1h25m32.508366126s) [upload=1235]: Starting upload Sep 6 14:53:56.468848823 (1h25m32.508421943s) [upload=1235]: Done starting upload Sep 6 14:53:56.468935940 (1h25m32.50850905s) [upload=1235]: Uploading segments... Sep 6 14:53:56.469043285 (1h25m32.508616405s) [upload=1235]: Got next segment. Inline: false Sep 6 14:53:56.469084605 (1h25m32.508657725s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 6 14:53:56.469089323 (1h25m32.508662443s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 6 14:53:57.220323838 (1h25m33.259896948s) [seg_pos={0 0}, upload=1235]: Begin upload segment... Sep 6 14:53:57.220365938 (1h25m33.259939058s) [seg_pos={0 0}, upload=1235]: Done begin upload segment. Sep 6 14:53:57.220387790 (1h25m33.2599609s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220450220 (1h25m33.26002333s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220798706 (1h25m33.260371826s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.220997257 (1h25m33.260570377s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221156230 (1h25m33.26072935s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221340182 (1h25m33.260913292s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221455262 (1h25m33.261028382s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221682848 (1h25m33.261255969s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.221742001 (1h25m33.261315122s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222029701 (1h25m33.261602822s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222037106 (1h25m33.261610216s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222072634 (1h25m33.261645744s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222318084 (1h25m33.261891204s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222416141 (1h25m33.261989261s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.222729972 (1h25m33.262303082s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223031659 (1h25m33.262604779s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.223331834 (1h25m33.262904954s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:53:57.225051665 (1h25m33.264624785s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Uploading piece... Sep 6 14:54:00.335150826 (1h25m36.374723946s) [upload=1235]: Next returned nil segment Sep 6 14:54:00.335172306 (1h25m36.374745426s) [upload=1235]: Waiting for error group managing segments... Sep 6 14:54:02.186624209 (1h25m38.226197329s) [noise=true , address=145.79.227.50:4399 , piece_id=ODMO2AOQZN74E5KLHCEO3W5C4JTDXQDTY6LC3ITZD3AXZ7SUDASA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.447764356 (1h25m38.487337476s) [noise=true , address=145.79.227.43:4191 , piece_id=QIDB74CTVMOXDSVM7M6PD3S2X3OFCRYMDOQ4GAFXVDBWN7GSCLJQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.633275506 (1h25m38.672848626s) [noise=true , address=145.79.227.49:4339 , piece_id=3N6NXNCLQ6V33Z5JNCU4KEDX7YMBFTDI6LDWSQDW5UVWZAUHBXRA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.661830437 (1h25m38.701403557s) [noise=true , address=45.91.116.5:4305 , piece_id=JUCXD6OSKCBMGEAMJ3JF77ZZOSDBEG7Z6UB4YHCVEZJQ24USSWYQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.712478913 (1h25m38.752052033s) [noise=true , address=145.79.227.2:4132 , piece_id=HZJQCG5IEQMHTUSJ7JHASG753T2T3JPVSLL6NB5K5Q4E7MTYNPSA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.713631719 (1h25m38.753204839s) [noise=true , address=145.79.227.45:4245 , piece_id=IZPPQNB2LIM54AGWV4Y4LCN3KEHOPBLNGAFQIPKSLSK57VRSB72A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.918359971 (1h25m38.957933091s) [noise=true , address=77.90.31.17:4777 , piece_id=OYKUX6LLT3ITCIBMUUEU5NP4RMZJFTXNMYCYFLDN3KMSC5SKUCLA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:02.920247252 (1h25m38.959820372s) [noise=true , address=77.90.31.13:4660 , piece_id=ATRJHIQBW4JZXSW3IRAN7ANXJUUWWYRK4QHWRRQJG2OB5UVTHG6A, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.447027056 (1h25m39.486600166s) [noise=true , address=45.91.116.9:4278 , piece_id=TTZPDIZC7MK62KRLYENJJ4HL4EXY6Q3DDVBONEWTDPMXDDHVCZEQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:03.695922293 (1h25m39.735495403s) [noise=true , address=145.79.227.41:4099 , piece_id=UHUUDJV2TVAV4WMHWAKO7RU22NUPRD5W3673CS6HIPFVXKQV2F4Q, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.247873844 (1h25m40.287446965s) [noise=true , address=145.79.227.37:4575 , piece_id=AHWNPDC5XCFYARVPKCFD3P3CHTOQMFVSOYQ474MU2UXAHRQQMPKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.274508982 (1h25m40.314082102s) [noise=true , address=145.79.227.3:4161 , piece_id=RI2T7FVXXYL66LLWGGV2USQ3NBJQFL5LLMK6CY2ZVF5UWKSSPZLQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.293548243 (1h25m40.333121373s) [noise=true , address=145.79.227.6:4265 , piece_id=TDGK4GXCCZLUTCLQO6AQ62LIDYCXYR6LS7VL7DQXJ23ZWJCS2TCQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487895030 (1h25m40.52746815s) [noise=true , address=77.90.31.16:4736 , piece_id=IIWPR6AQ7QYE2K4ZXIUYAVOTPXJTLC5ILNVYQHQYKDO2HG3CUNPA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: <nil> Sep 6 14:54:04.487923574 (1h25m40.527496694s) [seg_pos={0 0}, upload=1235]: Segment reached optimal threshold of 14 pieces. Sep 6 14:54:04.488050407 (1h25m40.527623527s) [noise=true , address=145.79.227.44:4196 , piece_id=FZZM3JCJAPUUICJMCW6QFPBQE53UJSNZTFQL47SB7CJ6J4WMDXEA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488071157 (1h25m40.527644277s) [noise=true , address=145.79.227.24:4588 , piece_id=AKKN35EWPYS4VLW5MZNIRW2XVUCPW6YJ5IEZEXTKPKGF5EWOQTGA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nMFTs5XGAKT6WGe7BRrAPNvuE1oCN7ZfiCfvWWwGDkCsKmQo5): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488113247 (1h25m40.527686367s) [noise=true , address=77.90.31.15:4714 , piece_id=IFUZFEOHEC4TJBZNP3BFEJYMIUNTWEVY34CU5LYXI5CL5SE3BIKQ, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488177922 (1h25m40.527751042s) [noise=true , address=145.79.227.4:4201 , piece_id=E5QDLVMPUWP4GJHRSQ6VF5F7H64EMOIMRSTVPNHTLFGJBCFF2VKA, seg_pos={0 0}, upload=1235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6): protocol: expected piece hash; context canceled; EOF Sep 6 14:54:04.488195304 (1h25m40.527768424s) [seg_pos={0 0}, upload=1235]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 6 14:54:04.488296448 (1h25m40.527869568s) [upload=1235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 6 14:54:04.488302009 (1h25m40.527875129s) [upload=1235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 6 14:54:05.506435449 (1h25m41.546008559s) [upload=1235]: Done uploading segments... Sep 6 14:54:05.506456279 (1h25m41.546029399s) [upload=1235]: Upload finished. err: <nil> Duration: 12.422695829s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)