Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X......X..X.....XX....X...X.....X.....XX] duration: 30.00337055s trace: https://jaeger.dev.storj.io/trace/2f62607aeab86c71 FAIL (2025-12-26 07:47:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: UCJJDR2Q3LROBKIGRKRTJTZV34ISNTHMG7GLIS7UGQF7YJ5ENSEA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: JENCIWUW2SCKOVT5PU7CT3XDTKBIACFB2IDTUJ7MD7UGKF6KZPJQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: BKBAY745JKSCGGGYHZCW3CJ6P2QKSR77FJEQ3F6DMPROPGLZO6PA) context deadline exceeded ======LOGS====== Dec 26 07:46:59.500346031 (496h17m59.371429076s) [upload=435608]: Starting upload Dec 26 07:46:59.500413710 (496h17m59.371496755s) [upload=435608]: Done starting upload Dec 26 07:46:59.500526805 (496h17m59.37160985s) [upload=435608]: Uploading segments... Dec 26 07:46:59.500641164 (496h17m59.371724199s) [upload=435608]: Got next segment. Inline: false Dec 26 07:46:59.500683494 (496h17m59.371766539s) [upload=435608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:46:59.500691048 (496h17m59.371774093s) [upload=435608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:46:59.742516015 (496h17m59.61359906s) [seg_pos={0 0}, upload=435608]: Begin upload segment... Dec 26 07:46:59.742592681 (496h17m59.613675716s) [seg_pos={0 0}, upload=435608]: Done begin upload segment. Dec 26 07:46:59.742653606 (496h17m59.613736641s) [noise=true , address=77.90.31.16:4735 , piece_id=2W2SXBFXOJH4WAZSJKBWFE2MHFBXGPC4TJL2ZQWIVXS3FBJQHXHQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.743844264 (496h17m59.614927299s) [noise=true , address=145.79.227.48:4302 , piece_id=U3C2MYQQ4IGXJKCFUHKBREOSIUKNGAFITJZOILQQF4H2ADNLATCQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.744515642 (496h17m59.615598687s) [noise=true , address=45.91.116.10:4082 , piece_id=JCXOJBU2IUWTF237I72TULKVGW52PO4Y4MW22Z2EDKUQCKU6FOFA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.745012057 (496h17m59.616095102s) [noise=true , address=45.91.116.9:4261 , piece_id=JENCIWUW2SCKOVT5PU7CT3XDTKBIACFB2IDTUJ7MD7UGKF6KZPJQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.745461102 (496h17m59.616544147s) [noise=true , address=145.79.227.24:4583 , piece_id=S746H363TP5DS2SV7VOIDFCYQZYQIVT3ELKEPLLXIP7AAIYNV7IA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.745976073 (496h17m59.617059108s) [noise=true , address=45.91.116.6:4973 , piece_id=OA2XNBRVPFE2B6I5OVYDWZOFBGETL25LW42G2U4SQ74MR7LQ2RUQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.747430483 (496h17m59.618513528s) [noise=true , address=145.79.227.52:4445 , piece_id=B5QBWYOOOYPJAMT2YGNJ4GR7PEF36OSAT6O6RHUUAGXA7R4IENYA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.748858121 (496h17m59.619941166s) [noise=true , address=145.79.227.11:4729 , piece_id=LB5PG4VSM3RQPWBXOLG5JEMGZKCQSBRTL6ZR22ZMT7P6O7QHF2JA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.750208341 (496h17m59.621291386s) [noise=true , address=45.91.116.8:4046 , piece_id=BKBAY745JKSCGGGYHZCW3CJ6P2QKSR77FJEQ3F6DMPROPGLZO6PA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.751369263 (496h17m59.622452308s) [noise=true , address=45.91.116.13:4156 , piece_id=ZWQYKQGH5GQZ3FR4E3KGUTE7BQD62H5DRSSZLM6HKSRDIBAKBGRQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.751698369 (496h17m59.622781414s) [noise=true , address=145.79.227.13:4802 , piece_id=42I4ZA2XR6AK7WOKKPRHBNWXXKFNHZKREZOOF6VYP5YOY7ZDXIDA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.753333823 (496h17m59.624416868s) [noise=true , address=145.79.227.33:4870 , piece_id=X7X7OIUE2UTRKOIE3B4X3TUR7HTJTDK6LKVX2VCJZKX7H4AFIHNA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.754320722 (496h17m59.625403767s) [noise=true , address=145.79.227.4:4185 , piece_id=KNPJLLAWA7UF2WKEOC7X7XYCUP2MW3FA3WHQPRUKWYULA5XT3FJA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.755810920 (496h17m59.626893955s) [noise=true , address=77.90.31.19:4844 , piece_id=772UP7KOQTBD6ZGYTSPXQ3MCVP64X3PCY3CYQOSQLP5RMNCQMEFQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.756576548 (496h17m59.627659593s) [noise=true , address=45.91.116.12:4225 , piece_id=UCJJDR2Q3LROBKIGRKRTJTZV34ISNTHMG7GLIS7UGQF7YJ5ENSEA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.757777805 (496h17m59.62886085s) [noise=true , address=145.79.227.51:4409 , piece_id=FXOLZRNC5QRTYRUIG4R2WBYNIBYXDGNA5H7R4ZFXZG2CRZ4OCY4Q, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:47:01.470306200 (496h18m1.341389245s) [upload=435608]: Next returned nil segment Dec 26 07:47:01.470331237 (496h18m1.341414282s) [upload=435608]: Waiting for error group managing segments... Dec 26 07:47:01.755785179 (496h18m1.626868234s) [noise=true , address=145.79.227.11:4729 , piece_id=LB5PG4VSM3RQPWBXOLG5JEMGZKCQSBRTL6ZR22ZMT7P6O7QHF2JA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.761053742 (496h18m1.632136787s) [noise=true , address=145.79.227.48:4302 , piece_id=U3C2MYQQ4IGXJKCFUHKBREOSIUKNGAFITJZOILQQF4H2ADNLATCQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.762628881 (496h18m1.633711926s) [noise=true , address=145.79.227.13:4802 , piece_id=42I4ZA2XR6AK7WOKKPRHBNWXXKFNHZKREZOOF6VYP5YOY7ZDXIDA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.763467428 (496h18m1.634550473s) [noise=true , address=145.79.227.24:4583 , piece_id=S746H363TP5DS2SV7VOIDFCYQZYQIVT3ELKEPLLXIP7AAIYNV7IA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.773924270 (496h18m1.645007315s) [noise=true , address=145.79.227.52:4445 , piece_id=B5QBWYOOOYPJAMT2YGNJ4GR7PEF36OSAT6O6RHUUAGXA7R4IENYA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.778328246 (496h18m1.649411291s) [noise=true , address=145.79.227.51:4409 , piece_id=FXOLZRNC5QRTYRUIG4R2WBYNIBYXDGNA5H7R4ZFXZG2CRZ4OCY4Q, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.788908192 (496h18m1.659991237s) [noise=true , address=145.79.227.4:4185 , piece_id=KNPJLLAWA7UF2WKEOC7X7XYCUP2MW3FA3WHQPRUKWYULA5XT3FJA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.014099153 (496h18m1.885182208s) [noise=true , address=45.91.116.8:4046 , piece_id=BKBAY745JKSCGGGYHZCW3CJ6P2QKSR77FJEQ3F6DMPROPGLZO6PA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.014136925 (496h18m1.88521997s) [noise=true , address=45.91.116.9:4261 , piece_id=JENCIWUW2SCKOVT5PU7CT3XDTKBIACFB2IDTUJ7MD7UGKF6KZPJQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.019282854 (496h18m1.890365899s) [noise=true , address=45.91.116.10:4082 , piece_id=JCXOJBU2IUWTF237I72TULKVGW52PO4Y4MW22Z2EDKUQCKU6FOFA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.027214217 (496h18m1.898297262s) [noise=true , address=77.90.31.19:4844 , piece_id=772UP7KOQTBD6ZGYTSPXQ3MCVP64X3PCY3CYQOSQLP5RMNCQMEFQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.102714321 (496h18m1.973797366s) [noise=true , address=45.91.116.12:4225 , piece_id=UCJJDR2Q3LROBKIGRKRTJTZV34ISNTHMG7GLIS7UGQF7YJ5ENSEA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.108561465 (496h18m1.97964451s) [noise=true , address=45.91.116.6:4973 , piece_id=OA2XNBRVPFE2B6I5OVYDWZOFBGETL25LW42G2U4SQ74MR7LQ2RUQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.108584319 (496h18m1.979667364s) [seg_pos={0 0}, upload=435608]: Segment reached optimal threshold of 13 pieces. Dec 26 07:47:02.108842851 (496h18m1.979925896s) [noise=true , address=77.90.31.16:4735 , piece_id=2W2SXBFXOJH4WAZSJKBWFE2MHFBXGPC4TJL2ZQWIVXS3FBJQHXHQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:02.108854744 (496h18m1.979937789s) [noise=true , address=77.90.31.16:4735 , piece_id=2W2SXBFXOJH4WAZSJKBWFE2MHFBXGPC4TJL2ZQWIVXS3FBJQHXHQ, seg_pos={0 0}, upload=435608]: Long tail cancellation due to optimal threshold Dec 26 07:47:02.108987055 (496h18m1.9800701s) [noise=true , address=145.79.227.33:4870 , piece_id=X7X7OIUE2UTRKOIE3B4X3TUR7HTJTDK6LKVX2VCJZKX7H4AFIHNA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:02.109003377 (496h18m1.980086412s) [noise=true , address=145.79.227.33:4870 , piece_id=X7X7OIUE2UTRKOIE3B4X3TUR7HTJTDK6LKVX2VCJZKX7H4AFIHNA, seg_pos={0 0}, upload=435608]: Long tail cancellation due to optimal threshold Dec 26 07:47:02.109208518 (496h18m1.980291563s) [noise=true , address=45.91.116.13:4156 , piece_id=ZWQYKQGH5GQZ3FR4E3KGUTE7BQD62H5DRSSZLM6HKSRDIBAKBGRQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:02.109218015 (496h18m1.98030106s) [noise=true , address=45.91.116.13:4156 , piece_id=ZWQYKQGH5GQZ3FR4E3KGUTE7BQD62H5DRSSZLM6HKSRDIBAKBGRQ, seg_pos={0 0}, upload=435608]: Long tail cancellation due to optimal threshold Dec 26 07:47:02.109238555 (496h18m1.9803216s) [seg_pos={0 0}, upload=435608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:47:02.109319278 (496h18m1.980402333s) [upload=435608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:47:02.109328005 (496h18m1.98041106s) [upload=435608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:47:02.397450669 (496h18m2.268533724s) [upload=435608]: Done uploading segments... Dec 26 07:47:02.397475306 (496h18m2.268558351s) [upload=435608]: Upload finished. err: <nil> Duration: 30.003405807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................X......X..X.....XX....X...X.....X.....X] duration: 30.011143452s trace: https://jaeger.dev.storj.io/trace/47d9959386134e9a FAIL (2025-12-26 07:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 07:46:26.392863497 (496h17m26.263946532s) [upload=435597]: Starting upload Dec 26 07:46:26.392924823 (496h17m26.264007858s) [upload=435597]: Done starting upload Dec 26 07:46:26.393025254 (496h17m26.264108299s) [upload=435597]: Uploading segments... Dec 26 07:46:26.393098223 (496h17m26.264181258s) [upload=435597]: Got next segment. Inline: false Dec 26 07:46:26.393131987 (496h17m26.264215072s) [upload=435597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:46:26.393137427 (496h17m26.264220462s) [upload=435597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:46:26.644476871 (496h17m26.515559916s) [seg_pos={0 0}, upload=435597]: Begin upload segment... Dec 26 07:46:26.644551112 (496h17m26.515634157s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6XNXSENYRKSRW5VIGVYSWPHIELFN3WALG2HZWSICU4K4LC57KFQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.644581260 (496h17m26.515664295s) [seg_pos={0 0}, upload=435597]: Done begin upload segment. Dec 26 07:46:26.644623500 (496h17m26.515706545s) [noise=true , address=145.79.227.35:4481 , piece_id=HAAUZMN65N5IRRTSPOQV3TLNAY77YDF5THIUFDEK3QJMIUCVWLDQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.645391492 (496h17m26.516474537s) [noise=true , address=45.91.116.5:4298 , piece_id=QZHMSVZUO6YCLWCIHFNIZOM67FDGJLCFPZINKMOHMKO5V5E3CN3A, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646214729 (496h17m26.517297774s) [noise=true , address=145.79.227.11:4732 , piece_id=EOF4ZKDGFL2TWNSEK3DBA2HH2FQUOQWVDIW4VOUJZFUWJ3VGTJRQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646499091 (496h17m26.517582137s) [noise=true , address=145.79.227.9:4366 , piece_id=C56TO7GYPSTODXWEN4FOGHRQE3PZLP4SRPTJ6AEDDBNQ3LCV3HLA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646828829 (496h17m26.517911874s) [noise=true , address=145.79.227.16:4910 , piece_id=WXOYNOFYDSKY4LLFSY4ZUIY4MHQYBG7DBOK3R423K7G4MZ4KB5IA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646842184 (496h17m26.517925219s) [noise=true , address=45.91.116.11:4117 , piece_id=Y7GTJIMDLOV3M4CINNFX6AAFLE642EEOLJCTHXLQFEQDZJUTSMBA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647107540 (496h17m26.518190585s) [noise=true , address=145.79.227.12:4765 , piece_id=M3HAVTTAWSA5AFDNR2JC4QY7IZKWIHMHTMAX56UHYYDRIJVEVQ2A, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647223731 (496h17m26.518306766s) [noise=true , address=45.91.116.7:4010 , piece_id=OUJK4NY7OKPW3PTVDMBYRV6XONDVDDNEMPTU3UW5RT4NNTWLC66Q, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647398523 (496h17m26.518481568s) [noise=true , address=45.91.116.10:4083 , piece_id=RQD4FMCYDKJKWP5CGW5LDWD3JCAQ6EDSDY4YZ6GX4MGADC5375FQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647679349 (496h17m26.518762394s) [noise=true , address=77.90.31.11:4555 , piece_id=JVD4ZUYEGOIOY2HHYAV7SDE7XDELVJBSSQDLQEYLWPKYGOVZTP4Q, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648000931 (496h17m26.519083986s) [noise=true , address=145.79.227.41:4087 , piece_id=GCGFTL6OQZCKUGQVK5TKH2QFECUCALUXU5VTN7OHJBFQB2NFLYIQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648331480 (496h17m26.519414515s) [noise=true , address=145.79.227.36:4518 , piece_id=AD7RL47BNBZKBBFRWVFF5KTY362L7J7T4H4MO7QNZNFUX7ELR5KQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648656530 (496h17m26.519739565s) [noise=true , address=45.91.116.6:4974 , piece_id=PWBT3CRWVCNHOJF4WFU4SRHOVPZLUCLESTCFGFPC7B472LNDZCWA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648996025 (496h17m26.52007907s) [noise=true , address=45.91.116.12:4225 , piece_id=5K4IBGSCVSG4CVZZAAHJEZR5I7TBJTSCEFCMVEY47DUI5QYPZFPQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.649233828 (496h17m26.520316873s) [noise=true , address=145.79.227.18:4006 , piece_id=L26J4SDWK2D5QJFW7C4XYBBZAZABOBJTQFSE2W7USBHMGYKE6J3A, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:28.364984237 (496h17m28.236067272s) [upload=435597]: Next returned nil segment Dec 26 07:46:28.364998245 (496h17m28.23608128s) [upload=435597]: Waiting for error group managing segments... Dec 26 07:46:28.650501751 (496h17m28.521584786s) [noise=true , address=145.79.227.18:4006 , piece_id=L26J4SDWK2D5QJFW7C4XYBBZAZABOBJTQFSE2W7USBHMGYKE6J3A, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.652863779 (496h17m28.523946814s) [noise=true , address=145.79.227.36:4518 , piece_id=AD7RL47BNBZKBBFRWVFF5KTY362L7J7T4H4MO7QNZNFUX7ELR5KQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.654667684 (496h17m28.525750729s) [noise=true , address=145.79.227.12:4765 , piece_id=M3HAVTTAWSA5AFDNR2JC4QY7IZKWIHMHTMAX56UHYYDRIJVEVQ2A, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.657515417 (496h17m28.528598452s) [noise=true , address=145.79.227.11:4732 , piece_id=EOF4ZKDGFL2TWNSEK3DBA2HH2FQUOQWVDIW4VOUJZFUWJ3VGTJRQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.657546757 (496h17m28.528629802s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6XNXSENYRKSRW5VIGVYSWPHIELFN3WALG2HZWSICU4K4LC57KFQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.672310940 (496h17m28.543393975s) [noise=true , address=145.79.227.16:4910 , piece_id=WXOYNOFYDSKY4LLFSY4ZUIY4MHQYBG7DBOK3R423K7G4MZ4KB5IA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.674878239 (496h17m28.545961294s) [noise=true , address=145.79.227.41:4087 , piece_id=GCGFTL6OQZCKUGQVK5TKH2QFECUCALUXU5VTN7OHJBFQB2NFLYIQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.681015725 (496h17m28.55209876s) [noise=true , address=145.79.227.35:4481 , piece_id=HAAUZMN65N5IRRTSPOQV3TLNAY77YDF5THIUFDEK3QJMIUCVWLDQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.682868172 (496h17m28.553951208s) [noise=true , address=145.79.227.9:4366 , piece_id=C56TO7GYPSTODXWEN4FOGHRQE3PZLP4SRPTJ6AEDDBNQ3LCV3HLA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.898229648 (496h17m28.769312693s) [noise=true , address=45.91.116.6:4974 , piece_id=PWBT3CRWVCNHOJF4WFU4SRHOVPZLUCLESTCFGFPC7B472LNDZCWA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.916470252 (496h17m28.787553307s) [noise=true , address=45.91.116.12:4225 , piece_id=5K4IBGSCVSG4CVZZAAHJEZR5I7TBJTSCEFCMVEY47DUI5QYPZFPQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.917138945 (496h17m28.78822199s) [noise=true , address=77.90.31.11:4555 , piece_id=JVD4ZUYEGOIOY2HHYAV7SDE7XDELVJBSSQDLQEYLWPKYGOVZTP4Q, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.984535378 (496h17m28.855618413s) [noise=true , address=45.91.116.11:4117 , piece_id=Y7GTJIMDLOV3M4CINNFX6AAFLE642EEOLJCTHXLQFEQDZJUTSMBA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.984561047 (496h17m28.855644092s) [seg_pos={0 0}, upload=435597]: Segment reached optimal threshold of 13 pieces. Dec 26 07:46:28.984790004 (496h17m28.855873039s) [noise=true , address=45.91.116.10:4083 , piece_id=RQD4FMCYDKJKWP5CGW5LDWD3JCAQ6EDSDY4YZ6GX4MGADC5375FQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 07:46:28.984798900 (496h17m28.855881935s) [noise=true , address=45.91.116.10:4083 , piece_id=RQD4FMCYDKJKWP5CGW5LDWD3JCAQ6EDSDY4YZ6GX4MGADC5375FQ, seg_pos={0 0}, upload=435597]: Long tail cancellation due to optimal threshold Dec 26 07:46:28.985133598 (496h17m28.856216683s) [noise=true , address=45.91.116.5:4298 , piece_id=QZHMSVZUO6YCLWCIHFNIZOM67FDGJLCFPZINKMOHMKO5V5E3CN3A, seg_pos={0 0}, upload=435597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39786->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39786->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39786->45.91.116.5:4298: use of closed network connection Dec 26 07:46:28.985145079 (496h17m28.856228115s) [noise=true , address=45.91.116.5:4298 , piece_id=QZHMSVZUO6YCLWCIHFNIZOM67FDGJLCFPZINKMOHMKO5V5E3CN3A, seg_pos={0 0}, upload=435597]: Long tail cancellation due to optimal threshold Dec 26 07:46:28.985308421 (496h17m28.856391466s) [noise=true , address=45.91.116.7:4010 , piece_id=OUJK4NY7OKPW3PTVDMBYRV6XONDVDDNEMPTU3UW5RT4NNTWLC66Q, seg_pos={0 0}, upload=435597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 07:46:28.985316586 (496h17m28.856399621s) [noise=true , address=45.91.116.7:4010 , piece_id=OUJK4NY7OKPW3PTVDMBYRV6XONDVDDNEMPTU3UW5RT4NNTWLC66Q, seg_pos={0 0}, upload=435597]: Long tail cancellation due to optimal threshold Dec 26 07:46:28.985338397 (496h17m28.856421432s) [seg_pos={0 0}, upload=435597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:46:28.985423600 (496h17m28.856506645s) [upload=435597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:46:28.985430182 (496h17m28.856513237s) [upload=435597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:46:29.294738116 (496h17m29.165821162s) [upload=435597]: Done uploading segments... Dec 26 07:46:29.294757394 (496h17m29.165840429s) [upload=435597]: Upload finished. err: <nil> Duration: 30.011156998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X......X..X.....XX....X...X.....X.....] duration: 5.766312255s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:46:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.766346439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X......X..X.....XX....X...X.....X....] duration: 5.729133821s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:45:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.729152126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X......X..X.....XX....X...X.....X...] duration: 5.759622005s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:44:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.759651311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................X......X..X.....XX....X...X.....X..] duration: 5.73062238s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:43:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.730648068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................X......X..X.....XX....X...X.....X.] duration: 5.956819977s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:42:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.956852891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................X......X..X.....XX....X...X.....X] duration: 30.004144679s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:41:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 30.004167382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.414622435s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:41:24 UTC) orbiter warning: context deadline exceeded Duration: 3.414651813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X......X..X.....XX....X...X.....] duration: 5.730870463s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:41:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 5.730905089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.514861388s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:40:29 UTC) orbiter warning: context deadline exceeded Duration: 3.514930709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.459891956s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:39:56 UTC) orbiter warning: context deadline exceeded Duration: 3.459926451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X......X..X.....XX....X...X....] duration: 5.749704635s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 5.749733911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X......X..X.....XX....X...X...] duration: 6.047907081s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:39:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 6.047950383s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.479325261s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:39:14 UTC) orbiter warning: context deadline exceeded Duration: 3.479355109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...............................X......X..X.....XX....X...X..] duration: 6.025912389s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:38:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 6.025940471s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.582318057s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:38:15 UTC) orbiter warning: context deadline exceeded Duration: 3.582342643s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [................................X......X..X.....XX....X...X.] duration: 5.992667066s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:37:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 5.992692795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.................................X......X..X.....XX....X...X] duration: 30.003716718s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:37:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 30.003739712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.621898895s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:37:00 UTC) orbiter warning: context deadline exceeded Duration: 3.621927251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.................................X......X..X.....XX....X...] duration: 6.038133772s trace: https://jaeger.dev.storj.io/trace/a199434bb55cc3 FAIL (2025-12-26 07:36:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA) context deadline exceeded ======LOGS====== Dec 26 07:34:47.498313588 (496h5m47.369396634s) [upload=435434]: Starting upload Dec 26 07:34:47.498522897 (496h5m47.369605942s) [upload=435434]: Done starting upload Dec 26 07:34:47.498559457 (496h5m47.369642492s) [upload=435434]: Uploading segments... Dec 26 07:34:47.498772612 (496h5m47.369855657s) [upload=435434]: Got next segment. Inline: false Dec 26 07:34:47.498816946 (496h5m47.369900002s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:34:47.498822246 (496h5m47.369905292s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:34:47.748609915 (496h5m47.61969296s) [seg_pos={0 0}, upload=435434]: Begin upload segment... Dec 26 07:34:47.748663897 (496h5m47.619746932s) [seg_pos={0 0}, upload=435434]: Done begin upload segment. Dec 26 07:34:47.748685468 (496h5m47.619768503s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.748770820 (496h5m47.619853856s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.749651487 (496h5m47.620734532s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.750737316 (496h5m47.621820351s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752016380 (496h5m47.623099415s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752414279 (496h5m47.623497314s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759039344 (496h5m47.630122379s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759962452 (496h5m47.631045487s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.762867172 (496h5m47.633950207s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.767567242 (496h5m47.638650277s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.770049198 (496h5m47.641132233s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.771085291 (496h5m47.642168336s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.772061480 (496h5m47.643144515s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.774886599 (496h5m47.645969634s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.777065078 (496h5m47.648148123s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.779011043 (496h5m47.650094078s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:49.485722139 (496h5m49.356805184s) [upload=435434]: Next returned nil segment Dec 26 07:34:49.485739772 (496h5m49.356822817s) [upload=435434]: Waiting for error group managing segments... Dec 26 07:34:49.773411387 (496h5m49.644494432s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.775960110 (496h5m49.647043155s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.776234473 (496h5m49.647317518s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.779711454 (496h5m49.650794499s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.780026243 (496h5m49.651109288s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.781227941 (496h5m49.652310976s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.791449244 (496h5m49.662532279s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.793561026 (496h5m49.664644061s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.798659624 (496h5m49.669742669s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.804813943 (496h5m49.675896978s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.808023555 (496h5m49.67910659s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.952928348 (496h5m49.824011383s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029011813 (496h5m49.900094848s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029032302 (496h5m49.900115338s) [seg_pos={0 0}, upload=435434]: Segment reached optimal threshold of 13 pieces. Dec 26 07:34:50.029247302 (496h5m49.900330337s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029258232 (496h5m49.900341277s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029436281 (496h5m49.900519326s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029446822 (496h5m49.900529867s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029513659 (496h5m49.900596724s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029523868 (496h5m49.900606923s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029545109 (496h5m49.900628154s) [seg_pos={0 0}, upload=435434]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:34:50.029620773 (496h5m49.900703818s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:34:50.029627406 (496h5m49.900710451s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:34:50.329686959 (496h5m50.200769994s) [upload=435434]: Done uploading segments... Dec 26 07:34:50.329777362 (496h5m50.200860407s) [upload=435434]: Upload finished. err: <nil> Duration: 6.038160904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.................................X......X..X.....XX....X..] duration: 6.106532685s trace: https://jaeger.dev.storj.io/trace/a199434bb55cc3 FAIL (2025-12-26 07:36:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA) context deadline exceeded ======LOGS====== Dec 26 07:34:47.498313588 (496h5m47.369396634s) [upload=435434]: Starting upload Dec 26 07:34:47.498522897 (496h5m47.369605942s) [upload=435434]: Done starting upload Dec 26 07:34:47.498559457 (496h5m47.369642492s) [upload=435434]: Uploading segments... Dec 26 07:34:47.498772612 (496h5m47.369855657s) [upload=435434]: Got next segment. Inline: false Dec 26 07:34:47.498816946 (496h5m47.369900002s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:34:47.498822246 (496h5m47.369905292s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:34:47.748609915 (496h5m47.61969296s) [seg_pos={0 0}, upload=435434]: Begin upload segment... Dec 26 07:34:47.748663897 (496h5m47.619746932s) [seg_pos={0 0}, upload=435434]: Done begin upload segment. Dec 26 07:34:47.748685468 (496h5m47.619768503s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.748770820 (496h5m47.619853856s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.749651487 (496h5m47.620734532s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.750737316 (496h5m47.621820351s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752016380 (496h5m47.623099415s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752414279 (496h5m47.623497314s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759039344 (496h5m47.630122379s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759962452 (496h5m47.631045487s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.762867172 (496h5m47.633950207s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.767567242 (496h5m47.638650277s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.770049198 (496h5m47.641132233s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.771085291 (496h5m47.642168336s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.772061480 (496h5m47.643144515s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.774886599 (496h5m47.645969634s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.777065078 (496h5m47.648148123s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.779011043 (496h5m47.650094078s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:49.485722139 (496h5m49.356805184s) [upload=435434]: Next returned nil segment Dec 26 07:34:49.485739772 (496h5m49.356822817s) [upload=435434]: Waiting for error group managing segments... Dec 26 07:34:49.773411387 (496h5m49.644494432s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.775960110 (496h5m49.647043155s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.776234473 (496h5m49.647317518s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.779711454 (496h5m49.650794499s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.780026243 (496h5m49.651109288s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.781227941 (496h5m49.652310976s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.791449244 (496h5m49.662532279s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.793561026 (496h5m49.664644061s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.798659624 (496h5m49.669742669s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.804813943 (496h5m49.675896978s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.808023555 (496h5m49.67910659s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.952928348 (496h5m49.824011383s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029011813 (496h5m49.900094848s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029032302 (496h5m49.900115338s) [seg_pos={0 0}, upload=435434]: Segment reached optimal threshold of 13 pieces. Dec 26 07:34:50.029247302 (496h5m49.900330337s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029258232 (496h5m49.900341277s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029436281 (496h5m49.900519326s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029446822 (496h5m49.900529867s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029513659 (496h5m49.900596724s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029523868 (496h5m49.900606923s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029545109 (496h5m49.900628154s) [seg_pos={0 0}, upload=435434]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:34:50.029620773 (496h5m49.900703818s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:34:50.029627406 (496h5m49.900710451s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:34:50.329686959 (496h5m50.200769994s) [upload=435434]: Done uploading segments... Dec 26 07:34:50.329777362 (496h5m50.200860407s) [upload=435434]: Upload finished. err: <nil> Duration: 6.106552664s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.48878831s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:35:56 UTC) orbiter warning: context deadline exceeded Duration: 3.488822535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.40575412s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:35:37 UTC) orbiter warning: context deadline exceeded Duration: 3.405783807s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.................................X......X..X.....XX....X.] duration: 5.952685114s trace: https://jaeger.dev.storj.io/trace/a199434bb55cc3 FAIL (2025-12-26 07:35:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA) context deadline exceeded ======LOGS====== Dec 26 07:34:47.498313588 (496h5m47.369396634s) [upload=435434]: Starting upload Dec 26 07:34:47.498522897 (496h5m47.369605942s) [upload=435434]: Done starting upload Dec 26 07:34:47.498559457 (496h5m47.369642492s) [upload=435434]: Uploading segments... Dec 26 07:34:47.498772612 (496h5m47.369855657s) [upload=435434]: Got next segment. Inline: false Dec 26 07:34:47.498816946 (496h5m47.369900002s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:34:47.498822246 (496h5m47.369905292s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:34:47.748609915 (496h5m47.61969296s) [seg_pos={0 0}, upload=435434]: Begin upload segment... Dec 26 07:34:47.748663897 (496h5m47.619746932s) [seg_pos={0 0}, upload=435434]: Done begin upload segment. Dec 26 07:34:47.748685468 (496h5m47.619768503s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.748770820 (496h5m47.619853856s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.749651487 (496h5m47.620734532s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.750737316 (496h5m47.621820351s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752016380 (496h5m47.623099415s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752414279 (496h5m47.623497314s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759039344 (496h5m47.630122379s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759962452 (496h5m47.631045487s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.762867172 (496h5m47.633950207s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.767567242 (496h5m47.638650277s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.770049198 (496h5m47.641132233s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.771085291 (496h5m47.642168336s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.772061480 (496h5m47.643144515s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.774886599 (496h5m47.645969634s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.777065078 (496h5m47.648148123s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.779011043 (496h5m47.650094078s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:49.485722139 (496h5m49.356805184s) [upload=435434]: Next returned nil segment Dec 26 07:34:49.485739772 (496h5m49.356822817s) [upload=435434]: Waiting for error group managing segments... Dec 26 07:34:49.773411387 (496h5m49.644494432s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.775960110 (496h5m49.647043155s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.776234473 (496h5m49.647317518s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.779711454 (496h5m49.650794499s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.780026243 (496h5m49.651109288s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.781227941 (496h5m49.652310976s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.791449244 (496h5m49.662532279s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.793561026 (496h5m49.664644061s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.798659624 (496h5m49.669742669s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.804813943 (496h5m49.675896978s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.808023555 (496h5m49.67910659s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.952928348 (496h5m49.824011383s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029011813 (496h5m49.900094848s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029032302 (496h5m49.900115338s) [seg_pos={0 0}, upload=435434]: Segment reached optimal threshold of 13 pieces. Dec 26 07:34:50.029247302 (496h5m49.900330337s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029258232 (496h5m49.900341277s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029436281 (496h5m49.900519326s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029446822 (496h5m49.900529867s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029513659 (496h5m49.900596724s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029523868 (496h5m49.900606923s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029545109 (496h5m49.900628154s) [seg_pos={0 0}, upload=435434]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:34:50.029620773 (496h5m49.900703818s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:34:50.029627406 (496h5m49.900710451s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:34:50.329686959 (496h5m50.200769994s) [upload=435434]: Done uploading segments... Dec 26 07:34:50.329777362 (496h5m50.200860407s) [upload=435434]: Upload finished. err: <nil> Duration: 5.952704682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.................................X......X..X.....XX....X] duration: 30.003565066s trace: https://jaeger.dev.storj.io/trace/a199434bb55cc3 FAIL (2025-12-26 07:35:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA) context deadline exceeded ======LOGS====== Dec 26 07:34:47.498313588 (496h5m47.369396634s) [upload=435434]: Starting upload Dec 26 07:34:47.498522897 (496h5m47.369605942s) [upload=435434]: Done starting upload Dec 26 07:34:47.498559457 (496h5m47.369642492s) [upload=435434]: Uploading segments... Dec 26 07:34:47.498772612 (496h5m47.369855657s) [upload=435434]: Got next segment. Inline: false Dec 26 07:34:47.498816946 (496h5m47.369900002s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:34:47.498822246 (496h5m47.369905292s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:34:47.748609915 (496h5m47.61969296s) [seg_pos={0 0}, upload=435434]: Begin upload segment... Dec 26 07:34:47.748663897 (496h5m47.619746932s) [seg_pos={0 0}, upload=435434]: Done begin upload segment. Dec 26 07:34:47.748685468 (496h5m47.619768503s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.748770820 (496h5m47.619853856s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.749651487 (496h5m47.620734532s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.750737316 (496h5m47.621820351s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752016380 (496h5m47.623099415s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.752414279 (496h5m47.623497314s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759039344 (496h5m47.630122379s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.759962452 (496h5m47.631045487s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.762867172 (496h5m47.633950207s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.767567242 (496h5m47.638650277s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.770049198 (496h5m47.641132233s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.771085291 (496h5m47.642168336s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.772061480 (496h5m47.643144515s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.774886599 (496h5m47.645969634s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.777065078 (496h5m47.648148123s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:47.779011043 (496h5m47.650094078s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Uploading piece... Dec 26 07:34:49.485722139 (496h5m49.356805184s) [upload=435434]: Next returned nil segment Dec 26 07:34:49.485739772 (496h5m49.356822817s) [upload=435434]: Waiting for error group managing segments... Dec 26 07:34:49.773411387 (496h5m49.644494432s) [noise=true , address=145.79.227.4:4187 , piece_id=5OIG6RTNRGXL4DWRSN7YRU5D2UCK3SLNME4TE33WG3VIZPNJAHPA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.775960110 (496h5m49.647043155s) [noise=true , address=145.79.227.33:4870 , piece_id=JNPGOYTGYGCH6N665L4JUD3UUAMM3FDJ2XN2WNKIC3HPYNGJWNVA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.776234473 (496h5m49.647317518s) [noise=true , address=145.79.227.24:4583 , piece_id=JSZQJQ37VOQWCDRKGQZCY72ESJNLLU3DOFBXEGMBDESBCZZUQGQQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.779711454 (496h5m49.650794499s) [noise=true , address=145.79.227.28:4727 , piece_id=DEQSAN7KIEHHIH42DZDP5RIML3TDJ2SJ3FM4BVCHNW4EV2AYYQ6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.780026243 (496h5m49.651109288s) [noise=true , address=145.79.227.41:4087 , piece_id=IJ3DDIRLTNSLMH3XH5M2TFFIZHJS2HBTZ7GKOXIINWBUDIGUII6A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.781227941 (496h5m49.652310976s) [noise=true , address=145.79.227.46:4603 , piece_id=RWRQGFKZ3G6IC3KTTMUM2U57AUVG2KOMP7PZVFLVE5MATJKSO2AA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.791449244 (496h5m49.662532279s) [noise=true , address=145.79.227.39:4014 , piece_id=QGNUSVLIFJGEQS2WVATSP3RGQ4TVHV6YGCN2BHV543MOVDF6462A, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.793561026 (496h5m49.664644061s) [noise=true , address=145.79.227.42:4122 , piece_id=XZTMHJ5BVVDSYMUZJ7FRTXUR3FUTROHKNXJTSLKJ3R3GNOMNB7XQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.798659624 (496h5m49.669742669s) [noise=true , address=145.79.227.2:4113 , piece_id=BTMNYZU4UPVXQHZRPQIYSPWZH5DPOJI4I3EUE3SWQ2XIV3OQEGFA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.804813943 (496h5m49.675896978s) [noise=true , address=145.79.227.47:4265 , piece_id=W3Q7W2YQSN7ETAOIJCJKUYK47ZDNG664HRDB25LBHPD2K5JTFAAA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.808023555 (496h5m49.67910659s) [noise=true , address=145.79.227.19:4077 , piece_id=33VVDLZLIZZ3H7UE42ANYZK4DSS7PCAXT54D23YIOBBWHVKJF5MA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:49.952928348 (496h5m49.824011383s) [noise=true , address=45.91.116.5:4300 , piece_id=E7HOZMRCKEG5GSOZJG52VEZYPF4WHK7OKRAR4D3RY244NDTJOICA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029011813 (496h5m49.900094848s) [noise=true , address=45.91.116.14:4191 , piece_id=JWNGBYTNDYSAP75EH5B3AS453276T2YUZNHGGHGSGAWVG2MMHUFQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: <nil> Dec 26 07:34:50.029032302 (496h5m49.900115338s) [seg_pos={0 0}, upload=435434]: Segment reached optimal threshold of 13 pieces. Dec 26 07:34:50.029247302 (496h5m49.900330337s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029258232 (496h5m49.900341277s) [noise=true , address=145.79.227.13:4804 , piece_id=LVHUIJV2G34EPHWM2Q5QWSTYORJ6OFBJUPMYXYC6NDYINOPGSUHQ, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029436281 (496h5m49.900519326s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029446822 (496h5m49.900529867s) [noise=true , address=45.91.116.12:4225 , piece_id=6WNLYONEJWE3QEP2Y3ODVMSUZNEELACWQYBXV47UVE3OQZ3L4CWA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029513659 (496h5m49.900596724s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 07:34:50.029523868 (496h5m49.900606923s) [noise=true , address=45.91.116.7:4012 , piece_id=RYXXTHEFCIRC2GEAZ6RENT24W5ASMLCGSGPISFS3NBHSEOAQ3BDA, seg_pos={0 0}, upload=435434]: Long tail cancellation due to optimal threshold Dec 26 07:34:50.029545109 (496h5m49.900628154s) [seg_pos={0 0}, upload=435434]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:34:50.029620773 (496h5m49.900703818s) [upload=435434]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:34:50.029627406 (496h5m49.900710451s) [upload=435434]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:34:50.329686959 (496h5m50.200769994s) [upload=435434]: Done uploading segments... Dec 26 07:34:50.329777362 (496h5m50.200860407s) [upload=435434]: Upload finished. err: <nil> Duration: 30.003597808s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 4.030346196s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:35:09 UTC) orbiter warning: context deadline exceeded Duration: 4.030398056s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.55307369s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:33:58 UTC) orbiter warning: context deadline exceeded Duration: 3.553100281s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X.................................X......X..X.....XX....] duration: 5.799016535s trace: https://jaeger.dev.storj.io/trace/29adabd164e6e93d FAIL (2025-12-26 07:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA) context deadline exceeded ======LOGS====== Dec 26 07:31:01.808469089 (496h2m1.679552134s) [upload=435378]: Starting upload Dec 26 07:31:01.808559312 (496h2m1.679642357s) [upload=435378]: Done starting upload Dec 26 07:31:01.808800100 (496h2m1.679883145s) [upload=435378]: Uploading segments... Dec 26 07:31:01.808855505 (496h2m1.67993855s) [upload=435378]: Got next segment. Inline: false Dec 26 07:31:01.808895972 (496h2m1.679979017s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:31:01.808917864 (496h2m1.680000909s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:31:02.055381082 (496h2m1.926464138s) [seg_pos={0 0}, upload=435378]: Begin upload segment... Dec 26 07:31:02.055446998 (496h2m1.926530043s) [seg_pos={0 0}, upload=435378]: Done begin upload segment. Dec 26 07:31:02.055487435 (496h2m1.92657048s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055510699 (496h2m1.926593745s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055631389 (496h2m1.926714434s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056205752 (496h2m1.927288807s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056626304 (496h2m1.927709349s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057049529 (496h2m1.928132575s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057468948 (496h2m1.928551993s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057995661 (496h2m1.929078706s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.058558713 (496h2m1.929641778s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059117166 (496h2m1.930200211s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059675209 (496h2m1.930758264s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061146600 (496h2m1.932229645s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061481518 (496h2m1.932564564s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.063245187 (496h2m1.934328232s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.064760472 (496h2m1.935843517s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.066612979 (496h2m1.937696034s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:03.789667415 (496h2m3.66075046s) [upload=435378]: Next returned nil segment Dec 26 07:31:03.789686592 (496h2m3.660769637s) [upload=435378]: Waiting for error group managing segments... Dec 26 07:31:04.071310471 (496h2m3.942393536s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.076348825 (496h2m3.94743189s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084922400 (496h2m3.956005445s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084944633 (496h2m3.956027678s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.095919681 (496h2m3.967002716s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.099656707 (496h2m3.970739752s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.120318630 (496h2m3.991401675s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.155872371 (496h2m4.026955416s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.191502206 (496h2m4.062585251s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.293257195 (496h2m4.16434026s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.302969759 (496h2m4.174052814s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.325559915 (496h2m4.19664297s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347441050 (496h2m4.218524095s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347464314 (496h2m4.21854736s) [seg_pos={0 0}, upload=435378]: Segment reached optimal threshold of 13 pieces. Dec 26 07:31:04.347647022 (496h2m4.218730067s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:42262->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 07:31:04.347657512 (496h2m4.218740567s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.347845560 (496h2m4.218928606s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.347853616 (496h2m4.218936661s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348065840 (496h2m4.219148885s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.348110935 (496h2m4.21919401s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348142255 (496h2m4.2192253s) [seg_pos={0 0}, upload=435378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:31:04.348236605 (496h2m4.21931965s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:31:04.348242035 (496h2m4.21932508s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:31:04.637651971 (496h2m4.508735036s) [upload=435378]: Done uploading segments... Dec 26 07:31:04.637691336 (496h2m4.508774401s) [upload=435378]: Upload finished. err: <nil> Duration: 5.799044438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...X.................................X......X..X.....XX...] duration: 5.768755873s trace: https://jaeger.dev.storj.io/trace/29adabd164e6e93d FAIL (2025-12-26 07:33:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA) context deadline exceeded ======LOGS====== Dec 26 07:31:01.808469089 (496h2m1.679552134s) [upload=435378]: Starting upload Dec 26 07:31:01.808559312 (496h2m1.679642357s) [upload=435378]: Done starting upload Dec 26 07:31:01.808800100 (496h2m1.679883145s) [upload=435378]: Uploading segments... Dec 26 07:31:01.808855505 (496h2m1.67993855s) [upload=435378]: Got next segment. Inline: false Dec 26 07:31:01.808895972 (496h2m1.679979017s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:31:01.808917864 (496h2m1.680000909s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:31:02.055381082 (496h2m1.926464138s) [seg_pos={0 0}, upload=435378]: Begin upload segment... Dec 26 07:31:02.055446998 (496h2m1.926530043s) [seg_pos={0 0}, upload=435378]: Done begin upload segment. Dec 26 07:31:02.055487435 (496h2m1.92657048s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055510699 (496h2m1.926593745s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055631389 (496h2m1.926714434s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056205752 (496h2m1.927288807s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056626304 (496h2m1.927709349s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057049529 (496h2m1.928132575s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057468948 (496h2m1.928551993s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057995661 (496h2m1.929078706s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.058558713 (496h2m1.929641778s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059117166 (496h2m1.930200211s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059675209 (496h2m1.930758264s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061146600 (496h2m1.932229645s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061481518 (496h2m1.932564564s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.063245187 (496h2m1.934328232s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.064760472 (496h2m1.935843517s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.066612979 (496h2m1.937696034s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:03.789667415 (496h2m3.66075046s) [upload=435378]: Next returned nil segment Dec 26 07:31:03.789686592 (496h2m3.660769637s) [upload=435378]: Waiting for error group managing segments... Dec 26 07:31:04.071310471 (496h2m3.942393536s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.076348825 (496h2m3.94743189s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084922400 (496h2m3.956005445s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084944633 (496h2m3.956027678s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.095919681 (496h2m3.967002716s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.099656707 (496h2m3.970739752s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.120318630 (496h2m3.991401675s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.155872371 (496h2m4.026955416s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.191502206 (496h2m4.062585251s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.293257195 (496h2m4.16434026s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.302969759 (496h2m4.174052814s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.325559915 (496h2m4.19664297s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347441050 (496h2m4.218524095s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347464314 (496h2m4.21854736s) [seg_pos={0 0}, upload=435378]: Segment reached optimal threshold of 13 pieces. Dec 26 07:31:04.347647022 (496h2m4.218730067s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:42262->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 07:31:04.347657512 (496h2m4.218740567s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.347845560 (496h2m4.218928606s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.347853616 (496h2m4.218936661s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348065840 (496h2m4.219148885s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.348110935 (496h2m4.21919401s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348142255 (496h2m4.2192253s) [seg_pos={0 0}, upload=435378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:31:04.348236605 (496h2m4.21931965s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:31:04.348242035 (496h2m4.21932508s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:31:04.637651971 (496h2m4.508735036s) [upload=435378]: Done uploading segments... Dec 26 07:31:04.637691336 (496h2m4.508774401s) [upload=435378]: Upload finished. err: <nil> Duration: 5.768782592s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.787827322s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:33:00 UTC) orbiter warning: context deadline exceeded Duration: 3.787855243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...X.................................X......X..X.....XX..] duration: 5.740764443s trace: https://jaeger.dev.storj.io/trace/29adabd164e6e93d FAIL (2025-12-26 07:32:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA) context deadline exceeded ======LOGS====== Dec 26 07:31:01.808469089 (496h2m1.679552134s) [upload=435378]: Starting upload Dec 26 07:31:01.808559312 (496h2m1.679642357s) [upload=435378]: Done starting upload Dec 26 07:31:01.808800100 (496h2m1.679883145s) [upload=435378]: Uploading segments... Dec 26 07:31:01.808855505 (496h2m1.67993855s) [upload=435378]: Got next segment. Inline: false Dec 26 07:31:01.808895972 (496h2m1.679979017s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:31:01.808917864 (496h2m1.680000909s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:31:02.055381082 (496h2m1.926464138s) [seg_pos={0 0}, upload=435378]: Begin upload segment... Dec 26 07:31:02.055446998 (496h2m1.926530043s) [seg_pos={0 0}, upload=435378]: Done begin upload segment. Dec 26 07:31:02.055487435 (496h2m1.92657048s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055510699 (496h2m1.926593745s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055631389 (496h2m1.926714434s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056205752 (496h2m1.927288807s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056626304 (496h2m1.927709349s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057049529 (496h2m1.928132575s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057468948 (496h2m1.928551993s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057995661 (496h2m1.929078706s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.058558713 (496h2m1.929641778s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059117166 (496h2m1.930200211s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059675209 (496h2m1.930758264s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061146600 (496h2m1.932229645s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061481518 (496h2m1.932564564s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.063245187 (496h2m1.934328232s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.064760472 (496h2m1.935843517s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.066612979 (496h2m1.937696034s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:03.789667415 (496h2m3.66075046s) [upload=435378]: Next returned nil segment Dec 26 07:31:03.789686592 (496h2m3.660769637s) [upload=435378]: Waiting for error group managing segments... Dec 26 07:31:04.071310471 (496h2m3.942393536s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.076348825 (496h2m3.94743189s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084922400 (496h2m3.956005445s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084944633 (496h2m3.956027678s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.095919681 (496h2m3.967002716s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.099656707 (496h2m3.970739752s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.120318630 (496h2m3.991401675s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.155872371 (496h2m4.026955416s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.191502206 (496h2m4.062585251s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.293257195 (496h2m4.16434026s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.302969759 (496h2m4.174052814s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.325559915 (496h2m4.19664297s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347441050 (496h2m4.218524095s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347464314 (496h2m4.21854736s) [seg_pos={0 0}, upload=435378]: Segment reached optimal threshold of 13 pieces. Dec 26 07:31:04.347647022 (496h2m4.218730067s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:42262->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 07:31:04.347657512 (496h2m4.218740567s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.347845560 (496h2m4.218928606s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.347853616 (496h2m4.218936661s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348065840 (496h2m4.219148885s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.348110935 (496h2m4.21919401s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348142255 (496h2m4.2192253s) [seg_pos={0 0}, upload=435378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:31:04.348236605 (496h2m4.21931965s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:31:04.348242035 (496h2m4.21932508s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:31:04.637651971 (496h2m4.508735036s) [upload=435378]: Done uploading segments... Dec 26 07:31:04.637691336 (496h2m4.508774401s) [upload=435378]: Upload finished. err: <nil> Duration: 5.740791193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X...X.................................X......X..X.....XX.] duration: 5.735403193s trace: https://jaeger.dev.storj.io/trace/29adabd164e6e93d FAIL (2025-12-26 07:31:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA) context deadline exceeded ======LOGS====== Dec 26 07:31:01.808469089 (496h2m1.679552134s) [upload=435378]: Starting upload Dec 26 07:31:01.808559312 (496h2m1.679642357s) [upload=435378]: Done starting upload Dec 26 07:31:01.808800100 (496h2m1.679883145s) [upload=435378]: Uploading segments... Dec 26 07:31:01.808855505 (496h2m1.67993855s) [upload=435378]: Got next segment. Inline: false Dec 26 07:31:01.808895972 (496h2m1.679979017s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:31:01.808917864 (496h2m1.680000909s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:31:02.055381082 (496h2m1.926464138s) [seg_pos={0 0}, upload=435378]: Begin upload segment... Dec 26 07:31:02.055446998 (496h2m1.926530043s) [seg_pos={0 0}, upload=435378]: Done begin upload segment. Dec 26 07:31:02.055487435 (496h2m1.92657048s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055510699 (496h2m1.926593745s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055631389 (496h2m1.926714434s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056205752 (496h2m1.927288807s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056626304 (496h2m1.927709349s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057049529 (496h2m1.928132575s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057468948 (496h2m1.928551993s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057995661 (496h2m1.929078706s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.058558713 (496h2m1.929641778s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059117166 (496h2m1.930200211s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059675209 (496h2m1.930758264s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061146600 (496h2m1.932229645s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061481518 (496h2m1.932564564s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.063245187 (496h2m1.934328232s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.064760472 (496h2m1.935843517s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.066612979 (496h2m1.937696034s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:03.789667415 (496h2m3.66075046s) [upload=435378]: Next returned nil segment Dec 26 07:31:03.789686592 (496h2m3.660769637s) [upload=435378]: Waiting for error group managing segments... Dec 26 07:31:04.071310471 (496h2m3.942393536s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.076348825 (496h2m3.94743189s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084922400 (496h2m3.956005445s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084944633 (496h2m3.956027678s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.095919681 (496h2m3.967002716s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.099656707 (496h2m3.970739752s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.120318630 (496h2m3.991401675s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.155872371 (496h2m4.026955416s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.191502206 (496h2m4.062585251s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.293257195 (496h2m4.16434026s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.302969759 (496h2m4.174052814s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.325559915 (496h2m4.19664297s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347441050 (496h2m4.218524095s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347464314 (496h2m4.21854736s) [seg_pos={0 0}, upload=435378]: Segment reached optimal threshold of 13 pieces. Dec 26 07:31:04.347647022 (496h2m4.218730067s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:42262->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 07:31:04.347657512 (496h2m4.218740567s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.347845560 (496h2m4.218928606s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.347853616 (496h2m4.218936661s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348065840 (496h2m4.219148885s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.348110935 (496h2m4.21919401s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348142255 (496h2m4.2192253s) [seg_pos={0 0}, upload=435378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:31:04.348236605 (496h2m4.21931965s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:31:04.348242035 (496h2m4.21932508s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:31:04.637651971 (496h2m4.508735036s) [upload=435378]: Done uploading segments... Dec 26 07:31:04.637691336 (496h2m4.508774401s) [upload=435378]: Upload finished. err: <nil> Duration: 5.735425585s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.744625477s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:31:49 UTC) orbiter warning: context deadline exceeded Duration: 3.744652859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X...X.................................X......X..X.....XX] duration: 30.003087341s trace: https://jaeger.dev.storj.io/trace/29adabd164e6e93d FAIL (2025-12-26 07:31:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: 25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA) context deadline exceeded ======LOGS====== Dec 26 07:31:01.808469089 (496h2m1.679552134s) [upload=435378]: Starting upload Dec 26 07:31:01.808559312 (496h2m1.679642357s) [upload=435378]: Done starting upload Dec 26 07:31:01.808800100 (496h2m1.679883145s) [upload=435378]: Uploading segments... Dec 26 07:31:01.808855505 (496h2m1.67993855s) [upload=435378]: Got next segment. Inline: false Dec 26 07:31:01.808895972 (496h2m1.679979017s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:31:01.808917864 (496h2m1.680000909s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:31:02.055381082 (496h2m1.926464138s) [seg_pos={0 0}, upload=435378]: Begin upload segment... Dec 26 07:31:02.055446998 (496h2m1.926530043s) [seg_pos={0 0}, upload=435378]: Done begin upload segment. Dec 26 07:31:02.055487435 (496h2m1.92657048s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055510699 (496h2m1.926593745s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.055631389 (496h2m1.926714434s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056205752 (496h2m1.927288807s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.056626304 (496h2m1.927709349s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057049529 (496h2m1.928132575s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057468948 (496h2m1.928551993s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.057995661 (496h2m1.929078706s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.058558713 (496h2m1.929641778s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059117166 (496h2m1.930200211s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.059675209 (496h2m1.930758264s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061146600 (496h2m1.932229645s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.061481518 (496h2m1.932564564s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.063245187 (496h2m1.934328232s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.064760472 (496h2m1.935843517s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:02.066612979 (496h2m1.937696034s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Uploading piece... Dec 26 07:31:03.789667415 (496h2m3.66075046s) [upload=435378]: Next returned nil segment Dec 26 07:31:03.789686592 (496h2m3.660769637s) [upload=435378]: Waiting for error group managing segments... Dec 26 07:31:04.071310471 (496h2m3.942393536s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2XHWGBKW3YOUHLQTSG6MX2OPASBB6OUGTGD2NQPFMFURZ32HOZA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.076348825 (496h2m3.94743189s) [noise=true , address=145.79.227.37:4565 , piece_id=L2XCSOCYS6DOVHMZTHRWUXVFEG6K5UXI67CJPG5JQWVSNFJVH26A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084922400 (496h2m3.956005445s) [noise=true , address=145.79.227.38:4977 , piece_id=FVR6VDIOCLYEJRME32B3IF5IZIWJFW3BUTFBIS7563OOS6BLAQHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.084944633 (496h2m3.956027678s) [noise=true , address=145.79.227.49:4338 , piece_id=EPZYOJL2GD7WTZWOKX7PJ43V5CZ37YXQBFXEAGODUPYA7LF6A6WQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.095919681 (496h2m3.967002716s) [noise=true , address=145.79.227.52:4446 , piece_id=OHDAUKKOZODAEBX7JB6WKAKF7PHMG2V2PR4GUWWHWTEG4GAKOMFQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.099656707 (496h2m3.970739752s) [noise=true , address=145.79.227.47:4265 , piece_id=MYVFYYKAFZRYHTUDAX4HCA6OL3WNSAMKA5SBNRJJELMK3ATV5Z5A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.120318630 (496h2m3.991401675s) [noise=true , address=145.79.227.16:4909 , piece_id=34LFMETTIS3C2LF2QOYC353UKSGAFSLQGS3QBR2CDXUV7Y5W72KA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.155872371 (496h2m4.026955416s) [noise=true , address=145.79.227.7:4293 , piece_id=FROYR23YSVP6AYTIJ3Y6R6PSALCOYRMRN3JHDVCP5K7MDIHHMNTQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.191502206 (496h2m4.062585251s) [noise=true , address=145.79.227.33:4870 , piece_id=DPZPGJ3FIZVCVHLCDAF27F4TOW4CLYPTETLO674Y5SOPBOXJ6NHQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.293257195 (496h2m4.16434026s) [noise=true , address=45.91.116.7:4011 , piece_id=25DUGNBIMDMIWVYX45U6SX6ZMGJ4ZQL2Q65BFYZ7CX53NKRIS77A, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.302969759 (496h2m4.174052814s) [noise=true , address=45.91.116.8:4045 , piece_id=Q2LSTOMLVSQNASGNR2ENGYPA4VDKI7FQT533KWMHPVF4O42GLSQQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.325559915 (496h2m4.19664297s) [noise=true , address=45.91.116.6:4974 , piece_id=PMVABYAPGS6AWLUOVIHUOSYGWS5CIXD67V7CGDQOTG2EI57PEYGQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347441050 (496h2m4.218524095s) [noise=true , address=45.91.116.5:4299 , piece_id=KYOGJVCQSAPGMOOYAK7FBTSUIB6MTMHTD4M5YECZ7BRCWQ6OQXDA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: <nil> Dec 26 07:31:04.347464314 (496h2m4.21854736s) [seg_pos={0 0}, upload=435378]: Segment reached optimal threshold of 13 pieces. Dec 26 07:31:04.347647022 (496h2m4.218730067s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:42262->77.90.31.19:4841: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 07:31:04.347657512 (496h2m4.218740567s) [noise=true , address=77.90.31.19:4841 , piece_id=V4YM7VYKYUR5KAVYPTYBUBY2VAQEJ6JU5PQO2IGNIRB2Z6JI6INA, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.347845560 (496h2m4.218928606s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.347853616 (496h2m4.218936661s) [noise=true , address=77.90.31.14:4661 , piece_id=C46M6OYXEHJ5FO4ZUFEMZSHI6MVYTOACF65DSMJEF5PAFA2QWZUQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348065840 (496h2m4.219148885s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 07:31:04.348110935 (496h2m4.21919401s) [noise=true , address=45.91.116.13:4155 , piece_id=L6JS3Y6SQMDLP6Z4ENZGWJ2VKCCG6SXYFKFFLCXHZPCHJSC4GJJQ, seg_pos={0 0}, upload=435378]: Long tail cancellation due to optimal threshold Dec 26 07:31:04.348142255 (496h2m4.2192253s) [seg_pos={0 0}, upload=435378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:31:04.348236605 (496h2m4.21931965s) [upload=435378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:31:04.348242035 (496h2m4.21932508s) [upload=435378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:31:04.637651971 (496h2m4.508735036s) [upload=435378]: Done uploading segments... Dec 26 07:31:04.637691336 (496h2m4.508774401s) [upload=435378]: Upload finished. err: <nil> Duration: 30.003098741s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.46795997s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:31:06 UTC) orbiter warning: context deadline exceeded Duration: 3.467991199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X...X.................................X......X..X.....X] duration: 30.003253391s trace: https://jaeger.dev.storj.io/trace/4c1818ccd44d1165 FAIL (2025-12-26 07:31:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 496h1m31.445009168s) [upload=435372]: Uploading segments... Dec 26 07:30:31.573970748 (496h1m31.445053793s) [upload=435372]: Got next segment. Inline: false Dec 26 07:30:31.574004262 (496h1m31.445087307s) [upload=435372]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:30:31.574009722 (496h1m31.445092757s) [upload=435372]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:30:31.817969241 (496h1m31.689052286s) [seg_pos={0 0}, upload=435372]: Begin upload segment... Dec 26 07:30:31.818039154 (496h1m31.689122189s) [seg_pos={0 0}, upload=435372]: Done begin upload segment. Dec 26 07:30:31.818101012 (496h1m31.689184057s) [noise=true , address=45.91.116.10:4081 , piece_id=3664JRQZHFZGBNWGFD25DM2L7AKDFLOF42I7IYMEQ3W44FSQYBMA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.818108947 (496h1m31.689191992s) [noise=true , address=145.79.227.36:4519 , piece_id=HAXRLHZBQI4ZYKNXBLWMM2OBRGWEVKRKBRDPQPGWC5VE2PR4HDQA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.819050280 (496h1m31.690133315s) [noise=true , address=145.79.227.4:4185 , piece_id=GUFDJRM2WOSKC53CP3XOVJAIGYNSCEKIH4BCA3ZIUQNP7OGCVQOA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.819300667 (496h1m31.690383702s) [noise=true , address=145.79.227.48:4302 , piece_id=AWRCCHL4KXFV52CZ7UNGF7TLZRWE6NVYGXGM6RUDGQ63OIARJGPA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.819723723 (496h1m31.690806758s) [noise=true , address=45.91.116.13:4156 , piece_id=VRS6SWVOTL4RD26PWIGPY6HML2K5UAVHBQ2QYLACIU3AVK7LVQ4Q, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.820120548 (496h1m31.691203583s) [noise=true , address=145.79.227.39:4013 , piece_id=WUJMPRKOTN7SNRIXPST2HVNWV3TBF73UHC6WOS673YFIMIOB6GPA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.820536279 (496h1m31.691619324s) [noise=true , address=145.79.227.28:4727 , piece_id=ZEYDJVRNWDGZW5LGNADBKJHFYMV5AHBU22YPKBQPWLAQGSHA6LVA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.820812916 (496h1m31.691895951s) [noise=true , address=145.79.227.45:4229 , piece_id=HTGEC5SL5CB7BH7YSE7725HNAKWAAUSWXS47FWX4GN5W7KQ4KH2Q, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.821211275 (496h1m31.69229432s) [noise=true , address=77.90.31.17:4770 , piece_id=CLI2IG52EIUFABSCQ7QVWBS3QJHBQF3YGESQCBYRM7SEKTNYKQCQ, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.821629542 (496h1m31.692712587s) [noise=true , address=45.91.116.12:4228 , piece_id=XUON4ZPDMQ6CPXMIVWCNTEEXANMTVSVZ6N3JLWJSEJY3ASMWV4KA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.821965170 (496h1m31.693048215s) [noise=true , address=145.79.227.42:4123 , piece_id=QVOC44Y56Z6UTMIM23E42HDG764Q6GFOFDV53KBE7IGGOGMQ2HHA, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.822259411 (496h1m31.693342446s) [noise=true , address=145.79.227.49:4337 , piece_id=YSJIBC636LQBVNUUK6J3D4GJCLX6UEOVIR56SPB4W43ILYQHJMMQ, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.822665854 (496h1m31.693748899s) [noise=true , address=45.91.116.6:4976 , piece_id=2SFV6KCSCMKYK75RKMN6T577V5HCZW2BJQEMKKOCBICSMGFTT65Q, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.823033705 (496h1m31.69411675s) [noise=true , address=145.79.227.41:4087 , piece_id=3DH63TBCQLKR2LYS2XUZ3ITU5FISAHXA4XCNJHHRZBOBJY6JGTNQ, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.823080945 (496h1m31.69416398s) [noise=true , address=145.79.227.18:4007 , piece_id=UDG36Q4PG4UKOPIO2EQJAHWSF76APAPPFYN52LUSDFLJ5ITASHOQ, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:31.823485215 (496h1m31.69456825s) [noise=true , address=145.79.227.5:4223 , piece_id=SWTG5K5BW4KMP26ENOBB6PWJE5V2DDNRBN2Q3EE2ZSAY2VAPX5PQ, seg_pos={0 0}, upload=435372]: Uploading piece... Dec 26 07:30:33.447572992 (496h1m33.318656037s) [upload=435372]: Next returned nil segment Dec 26 07:30:33.447590135 (496h1m33.31867318s) [upload=435372]: Waiting for error group managing segments... Dec 26 07:30:33.830111489 (496h1m33.701194534s) [noise=true , address=145.79.227.49:4337 , piece_id=YSJIBC636LQBVNUUK6J3D4GJCLX6UEOVIR56SPB4W43ILYQHJMMQ, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.830445445 (496h1m33.70152849s) [noise=true , address=145.79.227.39:4013 , piece_id=WUJMPRKOTN7SNRIXPST2HVNWV3TBF73UHC6WOS673YFIMIOB6GPA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.831286056 (496h1m33.702369091s) [noise=true , address=145.79.227.18:4007 , piece_id=UDG36Q4PG4UKOPIO2EQJAHWSF76APAPPFYN52LUSDFLJ5ITASHOQ, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.832943783 (496h1m33.704026828s) [noise=true , address=145.79.227.41:4087 , piece_id=3DH63TBCQLKR2LYS2XUZ3ITU5FISAHXA4XCNJHHRZBOBJY6JGTNQ, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.835044212 (496h1m33.706127247s) [noise=true , address=145.79.227.36:4519 , piece_id=HAXRLHZBQI4ZYKNXBLWMM2OBRGWEVKRKBRDPQPGWC5VE2PR4HDQA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.836353976 (496h1m33.707437021s) [noise=true , address=145.79.227.42:4123 , piece_id=QVOC44Y56Z6UTMIM23E42HDG764Q6GFOFDV53KBE7IGGOGMQ2HHA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.845391605 (496h1m33.71647465s) [noise=true , address=145.79.227.45:4229 , piece_id=HTGEC5SL5CB7BH7YSE7725HNAKWAAUSWXS47FWX4GN5W7KQ4KH2Q, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.845938817 (496h1m33.717021862s) [noise=true , address=145.79.227.48:4302 , piece_id=AWRCCHL4KXFV52CZ7UNGF7TLZRWE6NVYGXGM6RUDGQ63OIARJGPA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.847355976 (496h1m33.718439021s) [noise=true , address=145.79.227.28:4727 , piece_id=ZEYDJVRNWDGZW5LGNADBKJHFYMV5AHBU22YPKBQPWLAQGSHA6LVA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.848365949 (496h1m33.719448994s) [noise=true , address=145.79.227.5:4223 , piece_id=SWTG5K5BW4KMP26ENOBB6PWJE5V2DDNRBN2Q3EE2ZSAY2VAPX5PQ, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:33.849769461 (496h1m33.720852496s) [noise=true , address=145.79.227.4:4185 , piece_id=GUFDJRM2WOSKC53CP3XOVJAIGYNSCEKIH4BCA3ZIUQNP7OGCVQOA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:34.071179223 (496h1m33.942262258s) [noise=true , address=45.91.116.12:4228 , piece_id=XUON4ZPDMQ6CPXMIVWCNTEEXANMTVSVZ6N3JLWJSEJY3ASMWV4KA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:34.124535282 (496h1m33.995618317s) [noise=true , address=45.91.116.6:4976 , piece_id=2SFV6KCSCMKYK75RKMN6T577V5HCZW2BJQEMKKOCBICSMGFTT65Q, seg_pos={0 0}, upload=435372]: Done uploading piece. err: <nil> Dec 26 07:30:34.124559588 (496h1m33.995642633s) [seg_pos={0 0}, upload=435372]: Segment reached optimal threshold of 13 pieces. Dec 26 07:30:34.124770250 (496h1m33.995853285s) [noise=true , address=45.91.116.13:4156 , piece_id=VRS6SWVOTL4RD26PWIGPY6HML2K5UAVHBQ2QYLACIU3AVK7LVQ4Q, seg_pos={0 0}, upload=435372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:46388->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:46388->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46388->45.91.116.13:4156: use of closed network connection Dec 26 07:30:34.124778906 (496h1m33.995861951s) [noise=true , address=45.91.116.13:4156 , piece_id=VRS6SWVOTL4RD26PWIGPY6HML2K5UAVHBQ2QYLACIU3AVK7LVQ4Q, seg_pos={0 0}, upload=435372]: Long tail cancellation due to optimal threshold Dec 26 07:30:34.124870370 (496h1m33.995953405s) [noise=true , address=77.90.31.17:4770 , piece_id=CLI2IG52EIUFABSCQ7QVWBS3QJHBQF3YGESQCBYRM7SEKTNYKQCQ, seg_pos={0 0}, upload=435372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:36664->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:36664->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36664->77.90.31.17:4770: use of closed network connection Dec 26 07:30:34.124880439 (496h1m33.995963474s) [noise=true , address=77.90.31.17:4770 , piece_id=CLI2IG52EIUFABSCQ7QVWBS3QJHBQF3YGESQCBYRM7SEKTNYKQCQ, seg_pos={0 0}, upload=435372]: Long tail cancellation due to optimal threshold Dec 26 07:30:34.125094507 (496h1m33.996177542s) [noise=true , address=45.91.116.10:4081 , piece_id=3664JRQZHFZGBNWGFD25DM2L7AKDFLOF42I7IYMEQ3W44FSQYBMA, seg_pos={0 0}, upload=435372]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 07:30:34.125101550 (496h1m33.996184595s) [noise=true , address=45.91.116.10:4081 , piece_id=3664JRQZHFZGBNWGFD25DM2L7AKDFLOF42I7IYMEQ3W44FSQYBMA, seg_pos={0 0}, upload=435372]: Long tail cancellation due to optimal threshold Dec 26 07:30:34.125118984 (496h1m33.996202029s) [seg_pos={0 0}, upload=435372]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:30:34.125209736 (496h1m33.996292771s) [upload=435372]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:30:34.125214826 (496h1m33.996297871s) [upload=435372]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:30:34.409551851 (496h1m34.280634896s) [upload=435372]: Done uploading segments... Dec 26 07:30:34.409572300 (496h1m34.280655345s) [upload=435372]: Upload finished. err: <nil> Duration: 30.00327888s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.408596271s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:30:01 UTC) orbiter warning: context deadline exceeded Duration: 3.408621209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X.................................X......X..X.....] duration: 5.701015061s trace: https://jaeger.dev.storj.io/trace/2f71ebb8e20e9a10 FAIL (2025-12-26 07:29:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 56m0.064998921s) [upload=435288]: Uploading segments... Dec 26 07:25:00.194045141 (495h56m0.065128176s) [upload=435288]: Got next segment. Inline: false Dec 26 07:25:00.194089206 (495h56m0.065172251s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:25:00.194094836 (495h56m0.065177871s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:25:00.438149387 (495h56m0.309232442s) [seg_pos={0 0}, upload=435288]: Begin upload segment... Dec 26 07:25:00.438196968 (495h56m0.309280003s) [seg_pos={0 0}, upload=435288]: Done begin upload segment. Dec 26 07:25:00.438230091 (495h56m0.309313126s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438242103 (495h56m0.309325138s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438713662 (495h56m0.309796697s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438722739 (495h56m0.309805794s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439071232 (495h56m0.310154267s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439094587 (495h56m0.310177622s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439385761 (495h56m0.310468796s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439483618 (495h56m0.310566653s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439738953 (495h56m0.310821989s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439887396 (495h56m0.310970441s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439985593 (495h56m0.311068628s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440254946 (495h56m0.311337981s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440510512 (495h56m0.311593558s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441016526 (495h56m0.312099562s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441368857 (495h56m0.312451892s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441659250 (495h56m0.312742285s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:02.069638575 (495h56m1.94072162s) [upload=435288]: Next returned nil segment Dec 26 07:25:02.069656750 (495h56m1.940739795s) [upload=435288]: Waiting for error group managing segments... Dec 26 07:25:02.466649771 (495h56m2.337732816s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467240055 (495h56m2.33832311s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467371636 (495h56m2.338454691s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467877909 (495h56m2.338960954s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.468706958 (495h56m2.339790003s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469484338 (495h56m2.340567383s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469528443 (495h56m2.340611488s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470141570 (495h56m2.341224635s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470798881 (495h56m2.341881957s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.478893916 (495h56m2.349976971s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.534318463 (495h56m2.405401568s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.713878306 (495h56m2.584961351s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802890640 (495h56m2.673973685s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802921188 (495h56m2.674004233s) [seg_pos={0 0}, upload=435288]: Segment reached optimal threshold of 13 pieces. Dec 26 07:25:02.803199818 (495h56m2.674282853s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection Dec 26 07:25:02.803209197 (495h56m2.674292242s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803352099 (495h56m2.674435144s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 07:25:02.803364793 (495h56m2.674447868s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803889963 (495h56m2.674973008s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection Dec 26 07:25:02.803899941 (495h56m2.674982987s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803924949 (495h56m2.675007994s) [seg_pos={0 0}, upload=435288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:25:02.804011855 (495h56m2.6750949s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:25:02.804017606 (495h56m2.675100641s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:25:03.094106092 (495h56m2.965189137s) [upload=435288]: Done uploading segments... Dec 26 07:25:03.094140578 (495h56m2.965223623s) [upload=435288]: Upload finished. err: <nil> Duration: 5.701039557s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.501820887s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:29:13 UTC) orbiter warning: context deadline exceeded Duration: 3.501846454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X.................................X......X..X....] duration: 6.050335156s trace: https://jaeger.dev.storj.io/trace/2f71ebb8e20e9a10 FAIL (2025-12-26 07:28:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 56m0.064998921s) [upload=435288]: Uploading segments... Dec 26 07:25:00.194045141 (495h56m0.065128176s) [upload=435288]: Got next segment. Inline: false Dec 26 07:25:00.194089206 (495h56m0.065172251s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:25:00.194094836 (495h56m0.065177871s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:25:00.438149387 (495h56m0.309232442s) [seg_pos={0 0}, upload=435288]: Begin upload segment... Dec 26 07:25:00.438196968 (495h56m0.309280003s) [seg_pos={0 0}, upload=435288]: Done begin upload segment. Dec 26 07:25:00.438230091 (495h56m0.309313126s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438242103 (495h56m0.309325138s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438713662 (495h56m0.309796697s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438722739 (495h56m0.309805794s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439071232 (495h56m0.310154267s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439094587 (495h56m0.310177622s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439385761 (495h56m0.310468796s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439483618 (495h56m0.310566653s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439738953 (495h56m0.310821989s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439887396 (495h56m0.310970441s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439985593 (495h56m0.311068628s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440254946 (495h56m0.311337981s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440510512 (495h56m0.311593558s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441016526 (495h56m0.312099562s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441368857 (495h56m0.312451892s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441659250 (495h56m0.312742285s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:02.069638575 (495h56m1.94072162s) [upload=435288]: Next returned nil segment Dec 26 07:25:02.069656750 (495h56m1.940739795s) [upload=435288]: Waiting for error group managing segments... Dec 26 07:25:02.466649771 (495h56m2.337732816s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467240055 (495h56m2.33832311s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467371636 (495h56m2.338454691s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467877909 (495h56m2.338960954s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.468706958 (495h56m2.339790003s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469484338 (495h56m2.340567383s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469528443 (495h56m2.340611488s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470141570 (495h56m2.341224635s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470798881 (495h56m2.341881957s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.478893916 (495h56m2.349976971s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.534318463 (495h56m2.405401568s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.713878306 (495h56m2.584961351s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802890640 (495h56m2.673973685s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802921188 (495h56m2.674004233s) [seg_pos={0 0}, upload=435288]: Segment reached optimal threshold of 13 pieces. Dec 26 07:25:02.803199818 (495h56m2.674282853s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection Dec 26 07:25:02.803209197 (495h56m2.674292242s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803352099 (495h56m2.674435144s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 07:25:02.803364793 (495h56m2.674447868s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803889963 (495h56m2.674973008s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection Dec 26 07:25:02.803899941 (495h56m2.674982987s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803924949 (495h56m2.675007994s) [seg_pos={0 0}, upload=435288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:25:02.804011855 (495h56m2.6750949s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:25:02.804017606 (495h56m2.675100641s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:25:03.094106092 (495h56m2.965189137s) [upload=435288]: Done uploading segments... Dec 26 07:25:03.094140578 (495h56m2.965223623s) [upload=435288]: Upload finished. err: <nil> Duration: 6.050355535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......X...X.................................X......X..X...] duration: 5.776951536s trace: https://jaeger.dev.storj.io/trace/2f71ebb8e20e9a10 FAIL (2025-12-26 07:28:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 56m0.064998921s) [upload=435288]: Uploading segments... Dec 26 07:25:00.194045141 (495h56m0.065128176s) [upload=435288]: Got next segment. Inline: false Dec 26 07:25:00.194089206 (495h56m0.065172251s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:25:00.194094836 (495h56m0.065177871s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:25:00.438149387 (495h56m0.309232442s) [seg_pos={0 0}, upload=435288]: Begin upload segment... Dec 26 07:25:00.438196968 (495h56m0.309280003s) [seg_pos={0 0}, upload=435288]: Done begin upload segment. Dec 26 07:25:00.438230091 (495h56m0.309313126s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438242103 (495h56m0.309325138s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438713662 (495h56m0.309796697s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438722739 (495h56m0.309805794s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439071232 (495h56m0.310154267s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439094587 (495h56m0.310177622s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439385761 (495h56m0.310468796s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439483618 (495h56m0.310566653s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439738953 (495h56m0.310821989s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439887396 (495h56m0.310970441s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439985593 (495h56m0.311068628s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440254946 (495h56m0.311337981s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440510512 (495h56m0.311593558s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441016526 (495h56m0.312099562s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441368857 (495h56m0.312451892s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441659250 (495h56m0.312742285s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:02.069638575 (495h56m1.94072162s) [upload=435288]: Next returned nil segment Dec 26 07:25:02.069656750 (495h56m1.940739795s) [upload=435288]: Waiting for error group managing segments... Dec 26 07:25:02.466649771 (495h56m2.337732816s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467240055 (495h56m2.33832311s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467371636 (495h56m2.338454691s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467877909 (495h56m2.338960954s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.468706958 (495h56m2.339790003s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469484338 (495h56m2.340567383s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469528443 (495h56m2.340611488s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470141570 (495h56m2.341224635s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470798881 (495h56m2.341881957s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.478893916 (495h56m2.349976971s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.534318463 (495h56m2.405401568s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.713878306 (495h56m2.584961351s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802890640 (495h56m2.673973685s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802921188 (495h56m2.674004233s) [seg_pos={0 0}, upload=435288]: Segment reached optimal threshold of 13 pieces. Dec 26 07:25:02.803199818 (495h56m2.674282853s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection Dec 26 07:25:02.803209197 (495h56m2.674292242s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803352099 (495h56m2.674435144s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 07:25:02.803364793 (495h56m2.674447868s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803889963 (495h56m2.674973008s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection Dec 26 07:25:02.803899941 (495h56m2.674982987s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803924949 (495h56m2.675007994s) [seg_pos={0 0}, upload=435288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:25:02.804011855 (495h56m2.6750949s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:25:02.804017606 (495h56m2.675100641s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:25:03.094106092 (495h56m2.965189137s) [upload=435288]: Done uploading segments... Dec 26 07:25:03.094140578 (495h56m2.965223623s) [upload=435288]: Upload finished. err: <nil> Duration: 5.776980622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.4862523s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:28:12 UTC) orbiter warning: context deadline exceeded Duration: 3.486269434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.46677899s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:27:40 UTC) orbiter warning: context deadline exceeded Duration: 3.466832452s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......X...X.................................X......X..X..] duration: 5.886215454s trace: https://jaeger.dev.storj.io/trace/2f71ebb8e20e9a10 FAIL (2025-12-26 07:27:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 56m0.064998921s) [upload=435288]: Uploading segments... Dec 26 07:25:00.194045141 (495h56m0.065128176s) [upload=435288]: Got next segment. Inline: false Dec 26 07:25:00.194089206 (495h56m0.065172251s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:25:00.194094836 (495h56m0.065177871s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:25:00.438149387 (495h56m0.309232442s) [seg_pos={0 0}, upload=435288]: Begin upload segment... Dec 26 07:25:00.438196968 (495h56m0.309280003s) [seg_pos={0 0}, upload=435288]: Done begin upload segment. Dec 26 07:25:00.438230091 (495h56m0.309313126s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438242103 (495h56m0.309325138s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438713662 (495h56m0.309796697s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438722739 (495h56m0.309805794s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439071232 (495h56m0.310154267s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439094587 (495h56m0.310177622s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439385761 (495h56m0.310468796s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439483618 (495h56m0.310566653s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439738953 (495h56m0.310821989s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439887396 (495h56m0.310970441s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439985593 (495h56m0.311068628s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440254946 (495h56m0.311337981s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440510512 (495h56m0.311593558s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441016526 (495h56m0.312099562s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441368857 (495h56m0.312451892s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441659250 (495h56m0.312742285s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:02.069638575 (495h56m1.94072162s) [upload=435288]: Next returned nil segment Dec 26 07:25:02.069656750 (495h56m1.940739795s) [upload=435288]: Waiting for error group managing segments... Dec 26 07:25:02.466649771 (495h56m2.337732816s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467240055 (495h56m2.33832311s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467371636 (495h56m2.338454691s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467877909 (495h56m2.338960954s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.468706958 (495h56m2.339790003s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469484338 (495h56m2.340567383s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469528443 (495h56m2.340611488s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470141570 (495h56m2.341224635s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470798881 (495h56m2.341881957s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.478893916 (495h56m2.349976971s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.534318463 (495h56m2.405401568s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.713878306 (495h56m2.584961351s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802890640 (495h56m2.673973685s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802921188 (495h56m2.674004233s) [seg_pos={0 0}, upload=435288]: Segment reached optimal threshold of 13 pieces. Dec 26 07:25:02.803199818 (495h56m2.674282853s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection Dec 26 07:25:02.803209197 (495h56m2.674292242s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803352099 (495h56m2.674435144s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 07:25:02.803364793 (495h56m2.674447868s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803889963 (495h56m2.674973008s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection Dec 26 07:25:02.803899941 (495h56m2.674982987s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803924949 (495h56m2.675007994s) [seg_pos={0 0}, upload=435288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:25:02.804011855 (495h56m2.6750949s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:25:02.804017606 (495h56m2.675100641s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:25:03.094106092 (495h56m2.965189137s) [upload=435288]: Done uploading segments... Dec 26 07:25:03.094140578 (495h56m2.965223623s) [upload=435288]: Upload finished. err: <nil> Duration: 5.886241143s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.447463523s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:26:53 UTC) orbiter warning: context deadline exceeded Duration: 3.447490485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.461439479s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:26:21 UTC) orbiter warning: context deadline exceeded Duration: 3.461463645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.......X...X.................................X......X..X.] duration: 5.711295323s trace: https://jaeger.dev.storj.io/trace/2f71ebb8e20e9a10 FAIL (2025-12-26 07:26:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 56m0.064998921s) [upload=435288]: Uploading segments... Dec 26 07:25:00.194045141 (495h56m0.065128176s) [upload=435288]: Got next segment. Inline: false Dec 26 07:25:00.194089206 (495h56m0.065172251s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:25:00.194094836 (495h56m0.065177871s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:25:00.438149387 (495h56m0.309232442s) [seg_pos={0 0}, upload=435288]: Begin upload segment... Dec 26 07:25:00.438196968 (495h56m0.309280003s) [seg_pos={0 0}, upload=435288]: Done begin upload segment. Dec 26 07:25:00.438230091 (495h56m0.309313126s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438242103 (495h56m0.309325138s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438713662 (495h56m0.309796697s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438722739 (495h56m0.309805794s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439071232 (495h56m0.310154267s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439094587 (495h56m0.310177622s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439385761 (495h56m0.310468796s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439483618 (495h56m0.310566653s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439738953 (495h56m0.310821989s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439887396 (495h56m0.310970441s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439985593 (495h56m0.311068628s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440254946 (495h56m0.311337981s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440510512 (495h56m0.311593558s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441016526 (495h56m0.312099562s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441368857 (495h56m0.312451892s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441659250 (495h56m0.312742285s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:02.069638575 (495h56m1.94072162s) [upload=435288]: Next returned nil segment Dec 26 07:25:02.069656750 (495h56m1.940739795s) [upload=435288]: Waiting for error group managing segments... Dec 26 07:25:02.466649771 (495h56m2.337732816s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467240055 (495h56m2.33832311s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467371636 (495h56m2.338454691s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467877909 (495h56m2.338960954s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.468706958 (495h56m2.339790003s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469484338 (495h56m2.340567383s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469528443 (495h56m2.340611488s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470141570 (495h56m2.341224635s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470798881 (495h56m2.341881957s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.478893916 (495h56m2.349976971s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.534318463 (495h56m2.405401568s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.713878306 (495h56m2.584961351s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802890640 (495h56m2.673973685s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802921188 (495h56m2.674004233s) [seg_pos={0 0}, upload=435288]: Segment reached optimal threshold of 13 pieces. Dec 26 07:25:02.803199818 (495h56m2.674282853s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection Dec 26 07:25:02.803209197 (495h56m2.674292242s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803352099 (495h56m2.674435144s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 07:25:02.803364793 (495h56m2.674447868s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803889963 (495h56m2.674973008s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection Dec 26 07:25:02.803899941 (495h56m2.674982987s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803924949 (495h56m2.675007994s) [seg_pos={0 0}, upload=435288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:25:02.804011855 (495h56m2.6750949s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:25:02.804017606 (495h56m2.675100641s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:25:03.094106092 (495h56m2.965189137s) [upload=435288]: Done uploading segments... Dec 26 07:25:03.094140578 (495h56m2.965223623s) [upload=435288]: Upload finished. err: <nil> Duration: 5.711325611s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.469844836s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:25:56 UTC) orbiter warning: context deadline exceeded Duration: 3.469873901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.......X...X.................................X......X..X] duration: 30.002053027s trace: https://jaeger.dev.storj.io/trace/2f71ebb8e20e9a10 FAIL (2025-12-26 07:25:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 56m0.064998921s) [upload=435288]: Uploading segments... Dec 26 07:25:00.194045141 (495h56m0.065128176s) [upload=435288]: Got next segment. Inline: false Dec 26 07:25:00.194089206 (495h56m0.065172251s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:25:00.194094836 (495h56m0.065177871s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:25:00.438149387 (495h56m0.309232442s) [seg_pos={0 0}, upload=435288]: Begin upload segment... Dec 26 07:25:00.438196968 (495h56m0.309280003s) [seg_pos={0 0}, upload=435288]: Done begin upload segment. Dec 26 07:25:00.438230091 (495h56m0.309313126s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438242103 (495h56m0.309325138s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438713662 (495h56m0.309796697s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.438722739 (495h56m0.309805794s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439071232 (495h56m0.310154267s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439094587 (495h56m0.310177622s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439385761 (495h56m0.310468796s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439483618 (495h56m0.310566653s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439738953 (495h56m0.310821989s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439887396 (495h56m0.310970441s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.439985593 (495h56m0.311068628s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440254946 (495h56m0.311337981s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.440510512 (495h56m0.311593558s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441016526 (495h56m0.312099562s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441368857 (495h56m0.312451892s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:00.441659250 (495h56m0.312742285s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Uploading piece... Dec 26 07:25:02.069638575 (495h56m1.94072162s) [upload=435288]: Next returned nil segment Dec 26 07:25:02.069656750 (495h56m1.940739795s) [upload=435288]: Waiting for error group managing segments... Dec 26 07:25:02.466649771 (495h56m2.337732816s) [noise=true , address=145.79.227.5:4221 , piece_id=QPYB2LK4N77QIGMLRMXRAGRQ637X2WZQXPY7TGGZGVGGZ2AU4XKQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467240055 (495h56m2.33832311s) [noise=true , address=145.79.227.36:4519 , piece_id=F3EYVQ6OE6P35QQA4M7NT6TUVKUWWOCTBSIJE2ZCLX7GZE7XWS7Q, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467371636 (495h56m2.338454691s) [noise=true , address=145.79.227.17:4971 , piece_id=BFCI6AC43GFTX5YTSKN377SFSZGXZUFJBGQXWTQDCBR3F6YXQ22A, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.467877909 (495h56m2.338960954s) [noise=true , address=145.79.227.18:4006 , piece_id=3XXWTNAUGF7BF5ZYIYYOP2462M6MNB4PU52QESYG5QKDB4EGRMNQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.468706958 (495h56m2.339790003s) [noise=true , address=145.79.227.6:4259 , piece_id=AQMNQTU2VWD6XSKBBXSF4J2UACLE7MBAGM4ETSFSAEPVTFDRU3GQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469484338 (495h56m2.340567383s) [noise=true , address=145.79.227.25:4619 , piece_id=33SO7DGFTQCWDZ33MTMY22U7MSYBVDHUJ5ND3VTTI4EYFHGVUVDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.469528443 (495h56m2.340611488s) [noise=true , address=145.79.227.14:4838 , piece_id=Z3QOPHPJ7CZ6PCY4SMSUD3VVIU3A4KCI2OEV7F632E7JO3N7DJQA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470141570 (495h56m2.341224635s) [noise=true , address=145.79.227.45:4229 , piece_id=O4DPMLC5RD56UD3KEF7AAEXRADADZFIMKKL6OU3ZEKVT26L63LCQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.470798881 (495h56m2.341881957s) [noise=true , address=145.79.227.48:4302 , piece_id=S7BW5YA23OJ2IOK6DXJLYN5KMOOBY25VGIYDYOXGIIML3TWP2VNA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.478893916 (495h56m2.349976971s) [noise=true , address=145.79.227.41:4087 , piece_id=LT7I4LUJDYUJD7VN4SG3WNNI6HHHY5H6ICXGDQ5UQ7VASIJF56YA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.534318463 (495h56m2.405401568s) [noise=true , address=145.79.227.13:4803 , piece_id=AXF43TFD6JYEMJTGFPC3YDHOWY64KIKXM2IXX52AHJHPJHAO3XIQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.713878306 (495h56m2.584961351s) [noise=true , address=45.91.116.7:4010 , piece_id=D77VZ7RXUJVIBM4OHEND3YYSBXNMYV7A3ISAZGEM2HIGHN5HB6OQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802890640 (495h56m2.673973685s) [noise=true , address=77.90.31.18:4808 , piece_id=2RHLMAVV4LTQDAI7XPO5Q6B4ACHZD57PS3TY3ZCCY2ZKYYJCSAMQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: <nil> Dec 26 07:25:02.802921188 (495h56m2.674004233s) [seg_pos={0 0}, upload=435288]: Segment reached optimal threshold of 13 pieces. Dec 26 07:25:02.803199818 (495h56m2.674282853s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50968->45.91.116.14:4192: use of closed network connection Dec 26 07:25:02.803209197 (495h56m2.674292242s) [noise=true , address=45.91.116.14:4192 , piece_id=SV3BQOCUFUXPEOH4R5PSTU7BS42R4V5ACGPYH5K6JYQQTKXPMFAA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803352099 (495h56m2.674435144s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 26 07:25:02.803364793 (495h56m2.674447868s) [noise=true , address=45.91.116.6:4975 , piece_id=QB3JIO2YZV7QTSXBVZXMQKKFXOQ54S6GKQQ24J2YW2KXLDY7EGDQ, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803889963 (495h56m2.674973008s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35208->45.91.116.9:4262: use of closed network connection Dec 26 07:25:02.803899941 (495h56m2.674982987s) [noise=true , address=45.91.116.9:4262 , piece_id=ZMHXVWVEI7LEMT5ONHE3OO326PQVN4D7FFX3KMNVKXXR57XJYXUA, seg_pos={0 0}, upload=435288]: Long tail cancellation due to optimal threshold Dec 26 07:25:02.803924949 (495h56m2.675007994s) [seg_pos={0 0}, upload=435288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:25:02.804011855 (495h56m2.6750949s) [upload=435288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:25:02.804017606 (495h56m2.675100641s) [upload=435288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:25:03.094106092 (495h56m2.965189137s) [upload=435288]: Done uploading segments... Dec 26 07:25:03.094140578 (495h56m2.965223623s) [upload=435288]: Upload finished. err: <nil> Duration: 30.002084776s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)