Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 28.580745372s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:57:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 28.580773516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XXX.XX........XX..X.X............XX...........X..X.X.......] duration: 5.769604252s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:57:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 5.769629181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 21.538635106s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:57:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 21.538663661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XXX.XX........XX..X.X............XX...........X..X.X......] duration: 6.042017536s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:56:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 6.042043175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 24.053529876s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:56:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 24.053556798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XXX.XX........XX..X.X............XX...........X..X.X.....] duration: 5.806562347s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:56:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 5.806594208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 15.185733281s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 15.185796241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 26.744206782s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:55:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 26.744222521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....XXX.XX........XX..X.X............XX...........X..X.X....] duration: 5.932451001s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:55:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 5.932494845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 30.002013308s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:55:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 30.002042014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 22.577612212s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:54:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 22.577630275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....XXX.XX........XX..X.X............XX...........X..X.X...] duration: 6.192139635s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 6.192168069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.775431713s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:54:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 25.775464576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....XXX.XX........XX..X.X............XX...........X..X.X..] duration: 5.994206966s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:53:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 5.994237424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 19.69106713s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:53:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 19.691088171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....XXX.XX........XX..X.X............XX...........X..X.X.] duration: 7.627646331s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:52:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 7.627664387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 31.449936941s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 31.449998559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X....XXX.XX........XX..X.X............XX...........X..X.X] duration: 30.003512715s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 30.003536179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 30.268169407s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 30.268185928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003306353s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 1m0.003332222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX..X....XXX.XX........XX..X.X............XX...........X..X.] duration: 9.671303264s trace: https://jaeger.dev.storj.io/trace/4dba546429053535 FAIL (2025-12-26 05:51:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 05:50:39.722244243 (494h21m39.593327278s) [upload=433905]: Got next segment. Inline: false Dec 26 05:50:39.722292595 (494h21m39.59337563s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:50:39.722298927 (494h21m39.593381972s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:50:39.974786057 (494h21m39.845869102s) [seg_pos={0 0}, upload=433905]: Begin upload segment... Dec 26 05:50:39.974839899 (494h21m39.845922944s) [seg_pos={0 0}, upload=433905]: Done begin upload segment. Dec 26 05:50:39.974869265 (494h21m39.84595231s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.974884634 (494h21m39.845967669s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975552166 (494h21m39.846635201s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975611428 (494h21m39.846694473s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976074030 (494h21m39.847157075s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976515279 (494h21m39.847598324s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976682678 (494h21m39.847765713s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976937503 (494h21m39.848020538s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977403220 (494h21m39.848486266s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977891911 (494h21m39.848974946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.978557709 (494h21m39.849640754s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979222374 (494h21m39.850305419s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979380045 (494h21m39.85046309s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979853065 (494h21m39.85093611s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980340193 (494h21m39.851423238s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980826308 (494h21m39.851909353s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:41.850932079 (494h21m41.722015125s) [upload=433905]: Next returned nil segment Dec 26 05:50:41.850946236 (494h21m41.722029271s) [upload=433905]: Waiting for error group managing segments... Dec 26 05:50:42.153402526 (494h21m42.024485561s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.190326576 (494h21m42.061409632s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.213192848 (494h21m42.084275883s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.232589392 (494h21m42.103672437s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.286933563 (494h21m42.158016608s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.605865777 (494h21m42.476948822s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.885107401 (494h21m42.756190436s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.887324614 (494h21m42.758407649s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.925891632 (494h21m42.796974667s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.951119721 (494h21m42.822202756s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.952489940 (494h21m42.823572975s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.961906170 (494h21m42.832989205s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972590395 (494h21m42.84367343s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972615672 (494h21m42.843698707s) [seg_pos={0 0}, upload=433905]: Segment reached optimal threshold of 13 pieces. Dec 26 05:50:42.972798231 (494h21m42.843881266s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection Dec 26 05:50:42.972804553 (494h21m42.843887588s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973034911 (494h21m42.844117946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: context canceled; context canceled Dec 26 05:50:42.973045602 (494h21m42.844128637s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973081630 (494h21m42.844164665s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection Dec 26 05:50:42.973095807 (494h21m42.844178852s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973117959 (494h21m42.844200994s) [seg_pos={0 0}, upload=433905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:50:42.973221376 (494h21m42.844304411s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:50:42.973226155 (494h21m42.84430919s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:50:43.270711029 (494h21m43.141794074s) [upload=433905]: Done uploading segments... Dec 26 05:50:43.270744633 (494h21m43.141827668s) [upload=433905]: Upload finished. err: <nil> Duration: 9.671324015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X....XXX.XX........XX..X.X............XX...........X..X] duration: 30.002031081s trace: https://jaeger.dev.storj.io/trace/4dba546429053535 FAIL (2025-12-26 05:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 05:50:39.722244243 (494h21m39.593327278s) [upload=433905]: Got next segment. Inline: false Dec 26 05:50:39.722292595 (494h21m39.59337563s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:50:39.722298927 (494h21m39.593381972s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:50:39.974786057 (494h21m39.845869102s) [seg_pos={0 0}, upload=433905]: Begin upload segment... Dec 26 05:50:39.974839899 (494h21m39.845922944s) [seg_pos={0 0}, upload=433905]: Done begin upload segment. Dec 26 05:50:39.974869265 (494h21m39.84595231s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.974884634 (494h21m39.845967669s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975552166 (494h21m39.846635201s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975611428 (494h21m39.846694473s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976074030 (494h21m39.847157075s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976515279 (494h21m39.847598324s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976682678 (494h21m39.847765713s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976937503 (494h21m39.848020538s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977403220 (494h21m39.848486266s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977891911 (494h21m39.848974946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.978557709 (494h21m39.849640754s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979222374 (494h21m39.850305419s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979380045 (494h21m39.85046309s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979853065 (494h21m39.85093611s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980340193 (494h21m39.851423238s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980826308 (494h21m39.851909353s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:41.850932079 (494h21m41.722015125s) [upload=433905]: Next returned nil segment Dec 26 05:50:41.850946236 (494h21m41.722029271s) [upload=433905]: Waiting for error group managing segments... Dec 26 05:50:42.153402526 (494h21m42.024485561s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.190326576 (494h21m42.061409632s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.213192848 (494h21m42.084275883s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.232589392 (494h21m42.103672437s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.286933563 (494h21m42.158016608s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.605865777 (494h21m42.476948822s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.885107401 (494h21m42.756190436s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.887324614 (494h21m42.758407649s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.925891632 (494h21m42.796974667s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.951119721 (494h21m42.822202756s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.952489940 (494h21m42.823572975s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.961906170 (494h21m42.832989205s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972590395 (494h21m42.84367343s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972615672 (494h21m42.843698707s) [seg_pos={0 0}, upload=433905]: Segment reached optimal threshold of 13 pieces. Dec 26 05:50:42.972798231 (494h21m42.843881266s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection Dec 26 05:50:42.972804553 (494h21m42.843887588s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973034911 (494h21m42.844117946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: context canceled; context canceled Dec 26 05:50:42.973045602 (494h21m42.844128637s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973081630 (494h21m42.844164665s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection Dec 26 05:50:42.973095807 (494h21m42.844178852s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973117959 (494h21m42.844200994s) [seg_pos={0 0}, upload=433905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:50:42.973221376 (494h21m42.844304411s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:50:42.973226155 (494h21m42.84430919s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:50:43.270711029 (494h21m43.141794074s) [upload=433905]: Done uploading segments... Dec 26 05:50:43.270744633 (494h21m43.141827668s) [upload=433905]: Upload finished. err: <nil> Duration: 30.002051098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..X....XXX.XX........XX..X.X............XX...........X..] duration: 30.001996045s trace: https://jaeger.dev.storj.io/trace/6e2bf4b25e4f67b4 FAIL (2025-12-26 05:50:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:47:53.117961297 (494h18m52.989044342s) [upload=433866]: Starting upload Dec 26 05:47:53.118081555 (494h18m52.98916459s) [upload=433866]: Done starting upload Dec 26 05:47:53.118124949 (494h18m52.989207984s) [upload=433866]: Uploading segments... Dec 26 05:47:53.118319048 (494h18m52.989402093s) [upload=433866]: Got next segment. Inline: false Dec 26 05:47:53.118387438 (494h18m52.989470483s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:47:53.118394361 (494h18m52.989477406s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:47:53.361835516 (494h18m53.232918561s) [seg_pos={0 0}, upload=433866]: Begin upload segment... Dec 26 05:47:53.361896141 (494h18m53.232979186s) [seg_pos={0 0}, upload=433866]: Done begin upload segment. Dec 26 05:47:53.361944524 (494h18m53.233027569s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.361959562 (494h18m53.233042607s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362685184 (494h18m53.233768219s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362711675 (494h18m53.23379472s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363150640 (494h18m53.234233685s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363591169 (494h18m53.234674214s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364023081 (494h18m53.235106116s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364316040 (494h18m53.235399085s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364965896 (494h18m53.236048942s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365206635 (494h18m53.23628968s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365673104 (494h18m53.236756149s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366176432 (494h18m53.237259467s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366670673 (494h18m53.237753718s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367081676 (494h18m53.238164721s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367454095 (494h18m53.23853714s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.368036294 (494h18m53.239119339s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:55.081468871 (494h18m54.952551906s) [upload=433866]: Next returned nil segment Dec 26 05:47:55.081487868 (494h18m54.952570903s) [upload=433866]: Waiting for error group managing segments... Dec 26 05:47:55.367653326 (494h18m55.238736371s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.368747661 (494h18m55.239830706s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.369426843 (494h18m55.240509889s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.371809802 (494h18m55.242892847s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.382771103 (494h18m55.253854138s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.387164329 (494h18m55.258247364s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.390940750 (494h18m55.262023795s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393284734 (494h18m55.264367769s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393717187 (494h18m55.264800222s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.421978431 (494h18m55.293061466s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.542565630 (494h18m55.413648665s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.609192529 (494h18m55.480275574s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750220822 (494h18m55.621303867s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750241021 (494h18m55.621324056s) [seg_pos={0 0}, upload=433866]: Segment reached optimal threshold of 13 pieces. Dec 26 05:47:55.750532576 (494h18m55.621615621s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750541502 (494h18m55.621624547s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.750884536 (494h18m55.621967571s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750891931 (494h18m55.621974966s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751002631 (494h18m55.622085666s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection Dec 26 05:47:55.751007601 (494h18m55.622090636s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751022178 (494h18m55.622105223s) [seg_pos={0 0}, upload=433866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:47:55.751088264 (494h18m55.622171309s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:47:55.751094225 (494h18m55.62217726s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:47:56.043014072 (494h18m55.914097117s) [upload=433866]: Done uploading segments... Dec 26 05:47:56.043039020 (494h18m55.914122055s) [upload=433866]: Upload finished. err: <nil> Duration: 30.002020952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..XX..X....XXX.XX........XX..X.X............XX...........X.] duration: 6.736179612s trace: https://jaeger.dev.storj.io/trace/6e2bf4b25e4f67b4 FAIL (2025-12-26 05:49:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:47:53.117961297 (494h18m52.989044342s) [upload=433866]: Starting upload Dec 26 05:47:53.118081555 (494h18m52.98916459s) [upload=433866]: Done starting upload Dec 26 05:47:53.118124949 (494h18m52.989207984s) [upload=433866]: Uploading segments... Dec 26 05:47:53.118319048 (494h18m52.989402093s) [upload=433866]: Got next segment. Inline: false Dec 26 05:47:53.118387438 (494h18m52.989470483s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:47:53.118394361 (494h18m52.989477406s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:47:53.361835516 (494h18m53.232918561s) [seg_pos={0 0}, upload=433866]: Begin upload segment... Dec 26 05:47:53.361896141 (494h18m53.232979186s) [seg_pos={0 0}, upload=433866]: Done begin upload segment. Dec 26 05:47:53.361944524 (494h18m53.233027569s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.361959562 (494h18m53.233042607s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362685184 (494h18m53.233768219s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362711675 (494h18m53.23379472s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363150640 (494h18m53.234233685s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363591169 (494h18m53.234674214s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364023081 (494h18m53.235106116s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364316040 (494h18m53.235399085s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364965896 (494h18m53.236048942s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365206635 (494h18m53.23628968s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365673104 (494h18m53.236756149s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366176432 (494h18m53.237259467s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366670673 (494h18m53.237753718s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367081676 (494h18m53.238164721s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367454095 (494h18m53.23853714s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.368036294 (494h18m53.239119339s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:55.081468871 (494h18m54.952551906s) [upload=433866]: Next returned nil segment Dec 26 05:47:55.081487868 (494h18m54.952570903s) [upload=433866]: Waiting for error group managing segments... Dec 26 05:47:55.367653326 (494h18m55.238736371s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.368747661 (494h18m55.239830706s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.369426843 (494h18m55.240509889s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.371809802 (494h18m55.242892847s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.382771103 (494h18m55.253854138s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.387164329 (494h18m55.258247364s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.390940750 (494h18m55.262023795s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393284734 (494h18m55.264367769s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393717187 (494h18m55.264800222s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.421978431 (494h18m55.293061466s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.542565630 (494h18m55.413648665s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.609192529 (494h18m55.480275574s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750220822 (494h18m55.621303867s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750241021 (494h18m55.621324056s) [seg_pos={0 0}, upload=433866]: Segment reached optimal threshold of 13 pieces. Dec 26 05:47:55.750532576 (494h18m55.621615621s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750541502 (494h18m55.621624547s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.750884536 (494h18m55.621967571s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750891931 (494h18m55.621974966s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751002631 (494h18m55.622085666s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection Dec 26 05:47:55.751007601 (494h18m55.622090636s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751022178 (494h18m55.622105223s) [seg_pos={0 0}, upload=433866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:47:55.751088264 (494h18m55.622171309s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:47:55.751094225 (494h18m55.62217726s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:47:56.043014072 (494h18m55.914097117s) [upload=433866]: Done uploading segments... Dec 26 05:47:56.043039020 (494h18m55.914122055s) [upload=433866]: Upload finished. err: <nil> Duration: 6.73620984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..XX..X....XXX.XX........XX..X.X............XX...........X] duration: 30.002065438s trace: https://jaeger.dev.storj.io/trace/6e2bf4b25e4f67b4 FAIL (2025-12-26 05:48:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:47:53.117961297 (494h18m52.989044342s) [upload=433866]: Starting upload Dec 26 05:47:53.118081555 (494h18m52.98916459s) [upload=433866]: Done starting upload Dec 26 05:47:53.118124949 (494h18m52.989207984s) [upload=433866]: Uploading segments... Dec 26 05:47:53.118319048 (494h18m52.989402093s) [upload=433866]: Got next segment. Inline: false Dec 26 05:47:53.118387438 (494h18m52.989470483s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:47:53.118394361 (494h18m52.989477406s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:47:53.361835516 (494h18m53.232918561s) [seg_pos={0 0}, upload=433866]: Begin upload segment... Dec 26 05:47:53.361896141 (494h18m53.232979186s) [seg_pos={0 0}, upload=433866]: Done begin upload segment. Dec 26 05:47:53.361944524 (494h18m53.233027569s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.361959562 (494h18m53.233042607s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362685184 (494h18m53.233768219s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362711675 (494h18m53.23379472s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363150640 (494h18m53.234233685s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363591169 (494h18m53.234674214s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364023081 (494h18m53.235106116s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364316040 (494h18m53.235399085s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364965896 (494h18m53.236048942s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365206635 (494h18m53.23628968s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365673104 (494h18m53.236756149s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366176432 (494h18m53.237259467s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366670673 (494h18m53.237753718s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367081676 (494h18m53.238164721s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367454095 (494h18m53.23853714s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.368036294 (494h18m53.239119339s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:55.081468871 (494h18m54.952551906s) [upload=433866]: Next returned nil segment Dec 26 05:47:55.081487868 (494h18m54.952570903s) [upload=433866]: Waiting for error group managing segments... Dec 26 05:47:55.367653326 (494h18m55.238736371s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.368747661 (494h18m55.239830706s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.369426843 (494h18m55.240509889s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.371809802 (494h18m55.242892847s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.382771103 (494h18m55.253854138s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.387164329 (494h18m55.258247364s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.390940750 (494h18m55.262023795s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393284734 (494h18m55.264367769s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393717187 (494h18m55.264800222s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.421978431 (494h18m55.293061466s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.542565630 (494h18m55.413648665s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.609192529 (494h18m55.480275574s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750220822 (494h18m55.621303867s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750241021 (494h18m55.621324056s) [seg_pos={0 0}, upload=433866]: Segment reached optimal threshold of 13 pieces. Dec 26 05:47:55.750532576 (494h18m55.621615621s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750541502 (494h18m55.621624547s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.750884536 (494h18m55.621967571s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750891931 (494h18m55.621974966s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751002631 (494h18m55.622085666s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection Dec 26 05:47:55.751007601 (494h18m55.622090636s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751022178 (494h18m55.622105223s) [seg_pos={0 0}, upload=433866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:47:55.751088264 (494h18m55.622171309s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:47:55.751094225 (494h18m55.62217726s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:47:56.043014072 (494h18m55.914097117s) [upload=433866]: Done uploading segments... Dec 26 05:47:56.043039020 (494h18m55.914122055s) [upload=433866]: Upload finished. err: <nil> Duration: 30.002090885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..XX..X....XXX.XX........XX..X.X............XX...........] duration: 6.058403576s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.058428464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X..XX..X....XXX.XX........XX..X.X............XX..........] duration: 13.881591504s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 13.881619959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..XX..X....XXX.XX........XX..X.X............XX.........] duration: 5.897198855s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:45:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 5.8972277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X..XX..X....XXX.XX........XX..X.X............XX........] duration: 5.793356352s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:44:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 5.793384826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....X..XX..X....XXX.XX........XX..X.X............XX.......] duration: 5.766681376s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 5.766706575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X....X..XX..X....XXX.XX........XX..X.X............XX......] duration: 6.044478557s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:42:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.044514015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X....X..XX..X....XXX.XX........XX..X.X............XX.....] duration: 6.115486797s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.115512477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..X....X..XX..X....XXX.XX........XX..X.X............XX....] duration: 6.007962763s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:41:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.007988713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..X....X..XX..X....XXX.XX........XX..X.X............XX...] duration: 6.171234492s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:39:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.171264568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..X....X..XX..X....XXX.XX........XX..X.X............XX..] duration: 6.790210897s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:39:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.790237807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X..X....X..XX..X....XXX.XX........XX..X.X............XX.] duration: 7.622034339s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 7.622062624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X..X....X..XX..X....XXX.XX........XX..X.X............XX] duration: 30.003006982s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 30.003035596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.....X..X....X..XX..X....XXX.XX........XX..X.X............X] duration: 30.003227059s trace: https://jaeger.dev.storj.io/trace/42ae78b30ce7fa51 FAIL (2025-12-26 05:37:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: KQUVKDOBCHWGVZTGDXASYWB7MTHMKT64AS7ITQNV62Q3YFM7DLSA) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: SVYHWWGUQWAGGJKITAVEOEWG7BF5BEE6H4KWMQWM243LA6OOCMTQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: CQ5ANM6MHOJUXMWVDXIZHF5IFUN5CQTQ5BNUO7HHWU5B7GZ242RA) context deadline exceeded ======LOGS====== Dec 26 05:37:05.146523348 (494h8m5.017606393s) [upload=433710]: Starting upload Dec 26 05:37:05.146608641 (494h8m5.017691686s) [upload=433710]: Done starting upload Dec 26 05:37:05.146738588 (494h8m5.017821633s) [upload=433710]: Uploading segments... Dec 26 05:37:05.146829171 (494h8m5.017912206s) [upload=433710]: Got next segment. Inline: false Dec 26 05:37:05.146864889 (494h8m5.017947924s) [upload=433710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:05.146869477 (494h8m5.017952512s) [upload=433710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:05.661283967 (494h8m5.532367012s) [seg_pos={0 0}, upload=433710]: Begin upload segment... Dec 26 05:37:05.661331587 (494h8m5.532414632s) [seg_pos={0 0}, upload=433710]: Done begin upload segment. Dec 26 05:37:05.661360813 (494h8m5.532443858s) [noise=true , address=145.79.227.14:4838 , piece_id=N545MNWVBQHE7EYKYA3CDCL2TDCLAZAUMX2FA2EUD72IQD7Q24HA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.661531638 (494h8m5.532614673s) [noise=true , address=145.79.227.41:4087 , piece_id=O5V4HU3VVHGQL5PLXBSQQSKQMM4W2KIKQ4KHVA23MRGDFFGBZ47Q, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.662118847 (494h8m5.533201882s) [noise=true , address=145.79.227.45:4230 , piece_id=KC3EEECGW2PBUGQ4G3RJNNB5RP45UCYZRF4K3CNZBIOWA7MYKAOA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.662332824 (494h8m5.533415859s) [noise=true , address=145.79.227.3:4150 , piece_id=SXAQLZ4FUR3LY7QZGYDLSHU7R5N2OHCNJLVOCMZCZLWZ6EKSG2YQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.662772260 (494h8m5.533855295s) [noise=true , address=145.79.227.16:4912 , piece_id=KQUVKDOBCHWGVZTGDXASYWB7MTHMKT64AS7ITQNV62Q3YFM7DLSA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.663100396 (494h8m5.534183441s) [noise=true , address=45.91.116.7:4010 , piece_id=2RNRLPHN3MOQZGDVQAUGKVEHANDGHBW5NAC6N44SMY4NM2VNXEZQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.663484717 (494h8m5.534567762s) [noise=true , address=145.79.227.2:4113 , piece_id=KR6HOUNQTJSPAQ7WALAOXHIQWPLMYG5FGLDDLCKHN7FJPT5ARGQA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.663904627 (494h8m5.534987662s) [noise=true , address=145.79.227.11:4732 , piece_id=RWLKT5OTKHU5R37R4SJVF5SOHNT4I62JH7AVGMMK4CUS3YPL4EGQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.664164732 (494h8m5.535247767s) [noise=true , address=45.91.116.6:4976 , piece_id=ZB7XT5YBLZHDLBM6J3D6463I6YJEUATQRR5L3WKO5MTODHEJJJIQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.664719739 (494h8m5.535802774s) [noise=true , address=45.91.116.13:4156 , piece_id=ORK757KCI4UDBQIH4DVDL4Y7DOTS7LZPNQGWQKTMV7FNB7ISDPMQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.664972921 (494h8m5.536055956s) [noise=true , address=45.91.116.5:4298 , piece_id=45OQYHXQBPRRWJP7V33WTGGF6NUQ3QW7PJNV3JGULAYUTSFJQMYQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.665388893 (494h8m5.536471929s) [noise=true , address=145.79.227.51:4410 , piece_id=SVYHWWGUQWAGGJKITAVEOEWG7BF5BEE6H4KWMQWM243LA6OOCMTQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.665769587 (494h8m5.536852623s) [noise=true , address=145.79.227.35:4483 , piece_id=3QQBLX6IJFUGV2MWNOFW5ZMZXEKAVMXDEYHVW3LZYEHAAJD3I4XQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.665982814 (494h8m5.537065849s) [noise=true , address=145.79.227.38:4979 , piece_id=C6RMD5QUHRTH7OHCNEPZ7AYO4OBFKNXX7TVC47K3DEJCYG4AVS7A, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.667210211 (494h8m5.538293246s) [noise=true , address=145.79.227.50:4373 , piece_id=OADA6V5VB7D5BBRGP5SBCAFZIJDUWXXCPOKVXJLAWIJTXAGHMQGA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.667633607 (494h8m5.538716652s) [noise=true , address=145.79.227.19:4079 , piece_id=CQ5ANM6MHOJUXMWVDXIZHF5IFUN5CQTQ5BNUO7HHWU5B7GZ242RA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:07.534472808 (494h8m7.405555843s) [upload=433710]: Next returned nil segment Dec 26 05:37:07.534488368 (494h8m7.405571403s) [upload=433710]: Waiting for error group managing segments... Dec 26 05:37:07.836953728 (494h8m7.708036763s) [noise=true , address=145.79.227.38:4979 , piece_id=C6RMD5QUHRTH7OHCNEPZ7AYO4OBFKNXX7TVC47K3DEJCYG4AVS7A, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.841536774 (494h8m7.712619819s) [noise=true , address=145.79.227.50:4373 , piece_id=OADA6V5VB7D5BBRGP5SBCAFZIJDUWXXCPOKVXJLAWIJTXAGHMQGA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.842255964 (494h8m7.713338999s) [noise=true , address=145.79.227.35:4483 , piece_id=3QQBLX6IJFUGV2MWNOFW5ZMZXEKAVMXDEYHVW3LZYEHAAJD3I4XQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.852591014 (494h8m7.723674059s) [noise=true , address=145.79.227.16:4912 , piece_id=KQUVKDOBCHWGVZTGDXASYWB7MTHMKT64AS7ITQNV62Q3YFM7DLSA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.902495280 (494h8m7.773578315s) [noise=true , address=45.91.116.7:4010 , piece_id=2RNRLPHN3MOQZGDVQAUGKVEHANDGHBW5NAC6N44SMY4NM2VNXEZQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.915281598 (494h8m7.786364633s) [noise=true , address=145.79.227.14:4838 , piece_id=N545MNWVBQHE7EYKYA3CDCL2TDCLAZAUMX2FA2EUD72IQD7Q24HA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.003739768 (494h8m7.874822813s) [noise=true , address=45.91.116.13:4156 , piece_id=ORK757KCI4UDBQIH4DVDL4Y7DOTS7LZPNQGWQKTMV7FNB7ISDPMQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.553718047 (494h8m8.424801092s) [noise=true , address=145.79.227.19:4079 , piece_id=CQ5ANM6MHOJUXMWVDXIZHF5IFUN5CQTQ5BNUO7HHWU5B7GZ242RA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.564025194 (494h8m8.435108229s) [noise=true , address=145.79.227.51:4410 , piece_id=SVYHWWGUQWAGGJKITAVEOEWG7BF5BEE6H4KWMQWM243LA6OOCMTQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.571642799 (494h8m8.442725834s) [noise=true , address=145.79.227.11:4732 , piece_id=RWLKT5OTKHU5R37R4SJVF5SOHNT4I62JH7AVGMMK4CUS3YPL4EGQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.599604072 (494h8m8.470687107s) [noise=true , address=145.79.227.41:4087 , piece_id=O5V4HU3VVHGQL5PLXBSQQSKQMM4W2KIKQ4KHVA23MRGDFFGBZ47Q, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.629248240 (494h8m8.500331285s) [noise=true , address=45.91.116.6:4976 , piece_id=ZB7XT5YBLZHDLBM6J3D6463I6YJEUATQRR5L3WKO5MTODHEJJJIQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.644664886 (494h8m8.515747921s) [noise=true , address=45.91.116.5:4298 , piece_id=45OQYHXQBPRRWJP7V33WTGGF6NUQ3QW7PJNV3JGULAYUTSFJQMYQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.644696757 (494h8m8.515779792s) [seg_pos={0 0}, upload=433710]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:08.644883352 (494h8m8.515966387s) [noise=true , address=145.79.227.45:4230 , piece_id=KC3EEECGW2PBUGQ4G3RJNNB5RP45UCYZRF4K3CNZBIOWA7MYKAOA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: write tcp 188.166.225.170:42170->145.79.227.45:4230: use of closed network connection; write tcp 188.166.225.170:42170->145.79.227.45:4230: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42170->145.79.227.45:4230: use of closed network connection Dec 26 05:37:08.644889764 (494h8m8.515972799s) [noise=true , address=145.79.227.45:4230 , piece_id=KC3EEECGW2PBUGQ4G3RJNNB5RP45UCYZRF4K3CNZBIOWA7MYKAOA, seg_pos={0 0}, upload=433710]: Long tail cancellation due to optimal threshold Dec 26 05:37:08.645016776 (494h8m8.516099811s) [noise=true , address=145.79.227.3:4150 , piece_id=SXAQLZ4FUR3LY7QZGYDLSHU7R5N2OHCNJLVOCMZCZLWZ6EKSG2YQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:08.645033818 (494h8m8.516116853s) [noise=true , address=145.79.227.3:4150 , piece_id=SXAQLZ4FUR3LY7QZGYDLSHU7R5N2OHCNJLVOCMZCZLWZ6EKSG2YQ, seg_pos={0 0}, upload=433710]: Long tail cancellation due to optimal threshold Dec 26 05:37:08.645438720 (494h8m8.516521765s) [noise=true , address=145.79.227.2:4113 , piece_id=KR6HOUNQTJSPAQ7WALAOXHIQWPLMYG5FGLDDLCKHN7FJPT5ARGQA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:08.645453327 (494h8m8.516536372s) [noise=true , address=145.79.227.2:4113 , piece_id=KR6HOUNQTJSPAQ7WALAOXHIQWPLMYG5FGLDDLCKHN7FJPT5ARGQA, seg_pos={0 0}, upload=433710]: Long tail cancellation due to optimal threshold Dec 26 05:37:08.645477052 (494h8m8.516560098s) [seg_pos={0 0}, upload=433710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:08.645591480 (494h8m8.516674515s) [upload=433710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:08.645598474 (494h8m8.516681519s) [upload=433710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:08.959316911 (494h8m8.830399956s) [upload=433710]: Done uploading segments... Dec 26 05:37:08.959339103 (494h8m8.830422148s) [upload=433710]: Upload finished. err: <nil> Duration: 30.003249151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.....X..X....X..XX..X....XXX.XX........XX..X.X............] duration: 5.842800857s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.842829071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.....X..X....X..XX..X....XXX.XX........XX..X.X...........] duration: 18.333355274s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:36:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 18.33338469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX.....X..X....X..XX..X....XXX.XX........XX..X.X..........] duration: 5.99417676s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:35:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.994211619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.........] duration: 5.78989822s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.789926874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX.....X..X....X..XX..X....XXX.XX........XX..X.X........] duration: 5.831073677s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.831103174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.......] duration: 25.635049205s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:33:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 25.635087248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X......] duration: 6.013638423s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:31:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 6.013666447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.....] duration: 22.061339044s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:31:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 22.061367307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X....] duration: 15.734063697s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:30:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 15.734096029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X...] duration: 25.610883137s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:30:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 25.610913104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X..] duration: 5.869269758s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:29:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.869294836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.] duration: 5.983939046s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.983966409s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)