Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 25.775431713s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:54:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 25.775464576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X....XXX.XX........XX..X.X............XX...........X..X.X..] duration: 5.994206966s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:53:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 5.994237424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 19.69106713s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:53:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 19.691088171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X....XXX.XX........XX..X.X............XX...........X..X.X.] duration: 7.627646331s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:52:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 7.627664387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 31.449936941s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 31.449998559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X....XXX.XX........XX..X.X............XX...........X..X.X] duration: 30.003512715s trace: https://jaeger.dev.storj.io/trace/310352d49e3f345b FAIL (2025-12-26 05:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q) context deadline exceeded ======LOGS====== Dec 26 05:51:44.374802851 (494h22m44.245885886s) [upload=433921]: Starting upload Dec 26 05:51:44.374873185 (494h22m44.24595622s) [upload=433921]: Done starting upload Dec 26 05:51:44.374935272 (494h22m44.246018318s) [upload=433921]: Uploading segments... Dec 26 05:51:44.375136075 (494h22m44.24621911s) [upload=433921]: Got next segment. Inline: false Dec 26 05:51:44.375173717 (494h22m44.246256752s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:51:44.375178255 (494h22m44.24626129s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:51:44.620413406 (494h22m44.491496451s) [seg_pos={0 0}, upload=433921]: Begin upload segment... Dec 26 05:51:44.620470435 (494h22m44.49155348s) [seg_pos={0 0}, upload=433921]: Done begin upload segment. Dec 26 05:51:44.620515381 (494h22m44.491598426s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620521903 (494h22m44.491604938s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.620975617 (494h22m44.492058652s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621328549 (494h22m44.492411594s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621355400 (494h22m44.492438445s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621664920 (494h22m44.492747955s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.621735153 (494h22m44.492818188s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622144723 (494h22m44.493227758s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622158559 (494h22m44.493241604s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.622642521 (494h22m44.493725566s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623033636 (494h22m44.494116681s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623155578 (494h22m44.494238613s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623449517 (494h22m44.494532553s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.623754258 (494h22m44.494837303s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624747079 (494h22m44.495830114s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:44.624984270 (494h22m44.496067315s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Uploading piece... Dec 26 05:51:46.334127681 (494h22m46.205210726s) [upload=433921]: Next returned nil segment Dec 26 05:51:46.334141697 (494h22m46.205224732s) [upload=433921]: Waiting for error group managing segments... Dec 26 05:51:46.613317006 (494h22m46.484400051s) [noise=true , address=145.79.227.46:4603 , piece_id=LRMVN77MT2SALLS6VBF3HTZP4EE7QEGNJJH7OOYXICLFAMDZEZQQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621635767 (494h22m46.492718802s) [noise=true , address=145.79.227.6:4257 , piece_id=2H6FIOUBQWATNBGTZLULXVST7LLSZXGRKHLAEXJPSUX2XUMMKLOA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.621750174 (494h22m46.492833209s) [noise=true , address=145.79.227.48:4302 , piece_id=HPF7ZIPOGJUPWTKWV5SRJUIK34QEGJJPTQXB5KYPRNHKZH4YHIUQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629082216 (494h22m46.500165261s) [noise=true , address=145.79.227.36:4518 , piece_id=2J7DQHU3VP36Y7T7ZKLK5SRIZAU6ANHXFJOGBVADLC62FKKMOH7A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.629163971 (494h22m46.500247016s) [noise=true , address=145.79.227.13:4804 , piece_id=XCYXEPQYTRPCVWLHSI5ILHVIQ43XCC5ID4T5JWEBUB2NL6DH3CUA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.637052222 (494h22m46.508135257s) [noise=true , address=145.79.227.11:4732 , piece_id=BVEGKNIFZPIRSOD2H55BEVHT4IIIBLAOLUVJFVMXDUCDD2UM6L4A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.640538651 (494h22m46.511621686s) [noise=true , address=145.79.227.19:4077 , piece_id=RY3WGRX7QUFWY4UB3IRCT6E5LAN5TS5VUPHFXHCBDVUBVWWGZ3AQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.647818884 (494h22m46.518901919s) [noise=true , address=145.79.227.17:4970 , piece_id=7M5IQJDOUHM6QH422QDWTH3N6XJOLTS77XBDN65DAEQUHLGUJDTA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.656834661 (494h22m46.527917706s) [noise=true , address=145.79.227.14:4838 , piece_id=Y2HNQT5CLA2AHIUKRL4T5U3PKW4PQO6K67WEPSEDMSQKV3YM5EFA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.796042488 (494h22m46.667125523s) [noise=true , address=77.90.31.14:4662 , piece_id=IFKX4ZALGPC6UPZKF3MGCHNTRI7VNFWAOSAJA6FCDAYGEEIKYV4Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.845662783 (494h22m46.716745828s) [noise=true , address=77.90.31.16:4736 , piece_id=FRP4VUOAQ7CGN65AOBHD5X5U3EOIDE63LJWQB4UIFTYYGFNNQ7MA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.846752559 (494h22m46.717835604s) [noise=true , address=45.91.116.13:4154 , piece_id=S4O4XGEXCWPIUEFBHAFKDYGU63UBXG2UZWJN2ONYRCS4XRWHHWDQ, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851907755 (494h22m46.7229908s) [noise=true , address=45.91.116.14:4189 , piece_id=6O5Z2PC3GBYTPGFQTRPR4HKQOE5FU242AKD2MQP7WYLO5SPWBZ3Q, seg_pos={0 0}, upload=433921]: Done uploading piece. err: <nil> Dec 26 05:51:46.851952099 (494h22m46.723035134s) [seg_pos={0 0}, upload=433921]: Segment reached optimal threshold of 13 pieces. Dec 26 05:51:46.852278711 (494h22m46.723361756s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852289292 (494h22m46.723372337s) [noise=true , address=45.91.116.5:4297 , piece_id=SJKTWNXCCOL7ATMDOH3NIG7A5BE3BRFL22TNCIBNLC47CIFJLP5A, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852315741 (494h22m46.723398776s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852331452 (494h22m46.723414487s) [noise=true , address=77.90.31.17:4773 , piece_id=AR77AUFXR4DLPUA4JLNCYGVV6DAYM3HMLHGE4VGJA56QGYFMUAXA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852544027 (494h22m46.723627062s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 26 05:51:46.852550208 (494h22m46.723633253s) [noise=true , address=77.90.31.10:4518 , piece_id=IPYVWDQFC6VKMVEIH4MGODXZ7DZNRQJIZDATY65BZUWLPUJXWARA, seg_pos={0 0}, upload=433921]: Long tail cancellation due to optimal threshold Dec 26 05:51:46.852567131 (494h22m46.723650166s) [seg_pos={0 0}, upload=433921]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:51:46.852647233 (494h22m46.723730268s) [upload=433921]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:46.852652323 (494h22m46.723735358s) [upload=433921]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:47.141478028 (494h22m47.012561063s) [upload=433921]: Done uploading segments... Dec 26 05:51:47.141496634 (494h22m47.012579669s) [upload=433921]: Upload finished. err: <nil> Duration: 30.003536179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 30.268169407s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 30.268185928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.003306353s trace: https://jaeger.dev.storj.io/trace/7e198d4c8fea54d2 FAIL (2025-12-26 05:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, Piece ID: SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q) context deadline exceeded; piecestore: (Node ID: 1nP8xospdMuDEh3em5K6iBnh8bHXqLhvHy4pHm9m8Lnj4kEH2X, Piece ID: XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA) context deadline exceeded; piecestore: (Node ID: 1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, Piece ID: ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A) context deadline exceeded; piecestore: (Node ID: 12QmYSEAKnkSLEEVc9E1uxtVigunKMFLk2cdu71Bse8Tqec3HyT, Piece ID: YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ) context deadline exceeded; piecestore: (Node ID: 1JsnHuhQhM4rWLTLEuSSHLNE24m7tSW9qMSf48j7yMrAVPWVyV, Piece ID: ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q) context deadline exceeded ======LOGS====== upload=433902]: Uploading piece... Dec 26 05:50:34.986155265 (494h21m34.8572383s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.986703610 (494h21m34.857786655s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987145380 (494h21m34.858228425s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.987628290 (494h21m34.858711335s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.988710731 (494h21m34.859793776s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.989894545 (494h21m34.86097759s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.991305081 (494h21m34.862388116s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.995796263 (494h21m34.866879359s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.996765439 (494h21m34.867848484s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.998107634 (494h21m34.869190679s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:34.999340442 (494h21m34.870423477s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:35.000540757 (494h21m34.871623812s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Uploading piece... Dec 26 05:50:36.360077064 (494h21m36.231160099s) [upload=433902]: Next returned nil segment Dec 26 05:50:36.360091642 (494h21m36.231174688s) [upload=433902]: Waiting for error group managing segments... Dec 26 05:50:36.688914299 (494h21m36.559997344s) [noise=true , address=64.49.13.140:4509 , piece_id=6TGWTOAVY4YVAKU6VEE2SWVAM264TKU6ZNMGPWH4TNGII3UYXFXQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.059812634 (494h21m36.930895669s) [noise=true , address=77.90.31.11:4579 , piece_id=SBCEX7UNWGNFF54SBP4AGRDV5IPG24FQJXJJ7EYQTBSZKOVZWU4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.119433874 (494h21m36.990516909s) [noise=true , address=45.91.116.14:4204 , piece_id=XV7FQGBJNWVRLOAFRUPOWON35JR5ESRJPTCWPUGYWGHJPSDVL4GA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.138718204 (494h21m37.00980125s) [noise=true , address=77.90.31.16:4755 , piece_id=ARBMDAKX4QBW6JG47KP2PUITCX7HKXCBO75THQPINM4LLHCKAV4Q, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.146647243 (494h21m37.017730278s) [noise=true , address=77.90.31.13:4653 , piece_id=EY2CWX54ONOWPO3WC33G5QSP5RMP2SUTBQLUOH5JXGDRUEPDHQ2A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.163283722 (494h21m37.034366757s) [noise=true , address=77.90.31.21:4893 , piece_id=GNEEZO2XGOKNODOUPZQ4NGKFE4PCX5LNUWXLV27F6X7GWZVSYNAA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.374956574 (494h21m37.246039609s) [noise=true , address=45.91.116.7:4033 , piece_id=ATPIXLMXGONSSLXPPVORSNC6LHPXI2V3UFTHM4VHA2FTY5ZM6H3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:37.777283355 (494h21m37.64836639s) [noise=true , address=77.90.31.12:4622 , piece_id=LTYYPK6ZQ3NXZPBKSV4OMZGCOXXV3N6L5A6MVODNR644D2ZUJX3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439025909 (494h21m38.310108944s) [noise=true , address=145.79.227.49:4355 , piece_id=IUDOHR73Y6OKFAQSNNSONALWAITRJ4POYRO2QD5LYZKOSTWY4MNA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:38.439599111 (494h21m38.310682166s) [noise=true , address=77.90.31.10:4522 , piece_id=YFKD7GO2FOSPLDXNFPK732QTLCLBQQASWBTMZ2LTLBGSK22PIXIQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.259154980 (494h21m39.130238015s) [noise=true , address=145.79.227.34:4936 , piece_id=EEA2ZPM44DX3WMI7ZJTDC7T5UIGED6TNT53QXZWGBFRMQUQSCXPQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:39.430935921 (494h21m39.302018956s) [noise=true , address=145.79.227.44:4212 , piece_id=ABBNLKRE4FM7RHF56V4IAQPKBCCCQLLOMFVMSJEWQUNFGGCI3PVQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:50:42.941071297 (494h21m42.812154352s) [noise=true , address=77.90.31.14:4675 , piece_id=FKB4O45OYICMELPGXB65QAUD2DRIFTSXZFKIA6TY26VFAP6AVY3A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719026850 (494h22m22.590109885s) [noise=true , address=145.79.227.33:4890 , piece_id=UKNB4ZZTYJGY4MDHOBPYUTHCP3ZDCMJB7O7AJZFYP2HFVIWLUOKQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: <nil> Dec 26 05:51:22.719060193 (494h22m22.590143228s) [seg_pos={0 0}, upload=433902]: Segment reached optimal threshold of 14 pieces. Dec 26 05:51:22.719233022 (494h22m22.590316067s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zU35MWWaaC1npkP5qGcWBXAkfxGsd5UsdkM2nwX9x7wUGCDbY): protocol: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46970->145.79.227.50:4381: use of closed network connection Dec 26 05:51:22.719238783 (494h22m22.590321828s) [noise=true , address=145.79.227.50:4381 , piece_id=H6Q47DIOTOWGN7QH2KWT4IH6KVDSB246S22753Z3UQGB26T6VXPA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719251677 (494h22m22.590334712s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB): protocol: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58398->145.79.227.30:4785: use of closed network connection Dec 26 05:51:22.719260194 (494h22m22.590343229s) [noise=true , address=145.79.227.30:4785 , piece_id=5KMLU4ATBN4YVQDYJKKQY4PEJNKWTO2WNX5GMGJM4F54B6YM2SFA, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719409408 (494h22m22.590492443s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V): protocol: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45984->145.79.227.40:4052: use of closed network connection Dec 26 05:51:22.719415469 (494h22m22.590498515s) [noise=true , address=145.79.227.40:4052 , piece_id=XRZYL7PT5VHY22N7CCKS3NTZKDDWE36ZU3SORF6ILM7ZMXDND63A, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719467057 (494h22m22.590550092s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fNrbSfErEQRnruJCJAVvkoNwQSPWnvNtxunHn6zrQegnNAkxA): protocol: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55966->145.79.227.52:4461: use of closed network connection Dec 26 05:51:22.719473931 (494h22m22.590556966s) [noise=true , address=145.79.227.52:4461 , piece_id=ZAAUDBGY6CC2JNEGWXVPCBAWXARHAJDZ3YPB6Y3NWEWCFDUFGDSQ, seg_pos={0 0}, upload=433902]: Long tail cancellation due to optimal threshold Dec 26 05:51:22.719491374 (494h22m22.590574409s) [seg_pos={0 0}, upload=433902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 05:51:22.719562539 (494h22m22.590645574s) [upload=433902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:51:22.719567228 (494h22m22.590650263s) [upload=433902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:51:23.008169117 (494h22m22.879252162s) [upload=433902]: Done uploading segments... Dec 26 05:51:23.008191560 (494h22m22.879274605s) [upload=433902]: Upload finished. err: <nil> Duration: 1m0.003332222s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX..X....XXX.XX........XX..X.X............XX...........X..X.] duration: 9.671303264s trace: https://jaeger.dev.storj.io/trace/4dba546429053535 FAIL (2025-12-26 05:51:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 05:50:39.722244243 (494h21m39.593327278s) [upload=433905]: Got next segment. Inline: false Dec 26 05:50:39.722292595 (494h21m39.59337563s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:50:39.722298927 (494h21m39.593381972s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:50:39.974786057 (494h21m39.845869102s) [seg_pos={0 0}, upload=433905]: Begin upload segment... Dec 26 05:50:39.974839899 (494h21m39.845922944s) [seg_pos={0 0}, upload=433905]: Done begin upload segment. Dec 26 05:50:39.974869265 (494h21m39.84595231s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.974884634 (494h21m39.845967669s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975552166 (494h21m39.846635201s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975611428 (494h21m39.846694473s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976074030 (494h21m39.847157075s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976515279 (494h21m39.847598324s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976682678 (494h21m39.847765713s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976937503 (494h21m39.848020538s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977403220 (494h21m39.848486266s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977891911 (494h21m39.848974946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.978557709 (494h21m39.849640754s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979222374 (494h21m39.850305419s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979380045 (494h21m39.85046309s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979853065 (494h21m39.85093611s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980340193 (494h21m39.851423238s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980826308 (494h21m39.851909353s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:41.850932079 (494h21m41.722015125s) [upload=433905]: Next returned nil segment Dec 26 05:50:41.850946236 (494h21m41.722029271s) [upload=433905]: Waiting for error group managing segments... Dec 26 05:50:42.153402526 (494h21m42.024485561s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.190326576 (494h21m42.061409632s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.213192848 (494h21m42.084275883s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.232589392 (494h21m42.103672437s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.286933563 (494h21m42.158016608s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.605865777 (494h21m42.476948822s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.885107401 (494h21m42.756190436s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.887324614 (494h21m42.758407649s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.925891632 (494h21m42.796974667s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.951119721 (494h21m42.822202756s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.952489940 (494h21m42.823572975s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.961906170 (494h21m42.832989205s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972590395 (494h21m42.84367343s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972615672 (494h21m42.843698707s) [seg_pos={0 0}, upload=433905]: Segment reached optimal threshold of 13 pieces. Dec 26 05:50:42.972798231 (494h21m42.843881266s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection Dec 26 05:50:42.972804553 (494h21m42.843887588s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973034911 (494h21m42.844117946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: context canceled; context canceled Dec 26 05:50:42.973045602 (494h21m42.844128637s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973081630 (494h21m42.844164665s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection Dec 26 05:50:42.973095807 (494h21m42.844178852s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973117959 (494h21m42.844200994s) [seg_pos={0 0}, upload=433905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:50:42.973221376 (494h21m42.844304411s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:50:42.973226155 (494h21m42.84430919s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:50:43.270711029 (494h21m43.141794074s) [upload=433905]: Done uploading segments... Dec 26 05:50:43.270744633 (494h21m43.141827668s) [upload=433905]: Upload finished. err: <nil> Duration: 9.671324015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.XX..X....XXX.XX........XX..X.X............XX...........X..X] duration: 30.002031081s trace: https://jaeger.dev.storj.io/trace/4dba546429053535 FAIL (2025-12-26 05:51:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 26 05:50:39.722244243 (494h21m39.593327278s) [upload=433905]: Got next segment. Inline: false Dec 26 05:50:39.722292595 (494h21m39.59337563s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:50:39.722298927 (494h21m39.593381972s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:50:39.974786057 (494h21m39.845869102s) [seg_pos={0 0}, upload=433905]: Begin upload segment... Dec 26 05:50:39.974839899 (494h21m39.845922944s) [seg_pos={0 0}, upload=433905]: Done begin upload segment. Dec 26 05:50:39.974869265 (494h21m39.84595231s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.974884634 (494h21m39.845967669s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975552166 (494h21m39.846635201s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.975611428 (494h21m39.846694473s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976074030 (494h21m39.847157075s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976515279 (494h21m39.847598324s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976682678 (494h21m39.847765713s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.976937503 (494h21m39.848020538s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977403220 (494h21m39.848486266s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.977891911 (494h21m39.848974946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.978557709 (494h21m39.849640754s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979222374 (494h21m39.850305419s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979380045 (494h21m39.85046309s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.979853065 (494h21m39.85093611s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980340193 (494h21m39.851423238s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:39.980826308 (494h21m39.851909353s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Uploading piece... Dec 26 05:50:41.850932079 (494h21m41.722015125s) [upload=433905]: Next returned nil segment Dec 26 05:50:41.850946236 (494h21m41.722029271s) [upload=433905]: Waiting for error group managing segments... Dec 26 05:50:42.153402526 (494h21m42.024485561s) [noise=true , address=77.90.31.17:4773 , piece_id=5LFXBEVHL7UGBLSY3NXHUJZMRBYX4M5IHYC5QEGEZ3CCW6FCTY6A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.190326576 (494h21m42.061409632s) [noise=true , address=45.91.116.10:4084 , piece_id=X2BQURPUB4DSHQFQ4UWF2FY2NCOOGO7WZV4Y2PEHKYG7FVNTEB2Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.213192848 (494h21m42.084275883s) [noise=true , address=45.91.116.6:4973 , piece_id=25L6FQZP62WUJL4ZLFLF76Z4BHMARGSHIS7S2DEOZD6OTW5E2UHA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.232589392 (494h21m42.103672437s) [noise=true , address=77.90.31.14:4663 , piece_id=RLJRNVVJF5WIUERHEEPZ436GHF4PEMCF6NBODKUKKVXJSVY4U7ZQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.286933563 (494h21m42.158016608s) [noise=true , address=77.90.31.10:4518 , piece_id=MDUWNHUHXC7LOJONYDS64BXSTEA3IYK4XBMZ4W7227GOMPE3YL4A, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.605865777 (494h21m42.476948822s) [noise=true , address=145.79.227.36:4517 , piece_id=SEAI2FQIML4KPFGH3UVEINJQ624WWWVCU6YNP7DTJOUAINRIDMAQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.885107401 (494h21m42.756190436s) [noise=true , address=145.79.227.29:4942 , piece_id=LH7PBV45CKIMI3L37EP5KMKK2DC37IWVCFQ2ERFEMSJUTYGEGUTQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.887324614 (494h21m42.758407649s) [noise=true , address=145.79.227.2:4115 , piece_id=HG7IGWFPUMSU5IUUPNZTW3AIQC6IK2GBQD72F4JU3SPSXCOGAHGQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.925891632 (494h21m42.796974667s) [noise=true , address=145.79.227.3:4149 , piece_id=TMHFKF3Y5KIXMNIKZ7TPF2XDEV2F4NH6Z2QITTHMJCDFBYVD5KDA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.951119721 (494h21m42.822202756s) [noise=true , address=145.79.227.19:4077 , piece_id=YUBMDSPFE5PGFW2RKWBOMGGMHPHFXF56GUKF34OE5T3ZDXLWIT4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.952489940 (494h21m42.823572975s) [noise=true , address=145.79.227.17:4970 , piece_id=GJSVBPOLQW6G56OG4VMXHJUPZBR2SMUV5MXF27CITYHNX2VRSZ4Q, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.961906170 (494h21m42.832989205s) [noise=true , address=145.79.227.32:4835 , piece_id=SMOJBVQLJGUHLRG2GQRCQACGRN4PM6IV2F3QZXRVOLGC3Y4ZYPWA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972590395 (494h21m42.84367343s) [noise=true , address=145.79.227.6:4257 , piece_id=GYKMPCKVXHXQOBWBG5PAWPTK2B5ZJA6JFFPKXHQKOYDEFS2OG3AQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: <nil> Dec 26 05:50:42.972615672 (494h21m42.843698707s) [seg_pos={0 0}, upload=433905]: Segment reached optimal threshold of 13 pieces. Dec 26 05:50:42.972798231 (494h21m42.843881266s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34192->145.79.227.35:4481: use of closed network connection Dec 26 05:50:42.972804553 (494h21m42.843887588s) [noise=true , address=145.79.227.35:4481 , piece_id=AQ2A65ODEYCTIXULEXIU5EG7DIY4R4A4GP5VYWM2CFJH2YY4DFXQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973034911 (494h21m42.844117946s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: context canceled; context canceled Dec 26 05:50:42.973045602 (494h21m42.844128637s) [noise=true , address=145.79.227.52:4446 , piece_id=42LCILMTRQPM4E2CG47B6T47IFB7MD4KG4QP2FNJRBKMJVCL5HSA, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973081630 (494h21m42.844164665s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43720->145.79.227.46:4603: use of closed network connection Dec 26 05:50:42.973095807 (494h21m42.844178852s) [noise=true , address=145.79.227.46:4603 , piece_id=QC5IBLJDUJE5FRI3S6I65ETUMIFGJKA2LYM72UJM7XRHR75B3ZIQ, seg_pos={0 0}, upload=433905]: Long tail cancellation due to optimal threshold Dec 26 05:50:42.973117959 (494h21m42.844200994s) [seg_pos={0 0}, upload=433905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:50:42.973221376 (494h21m42.844304411s) [upload=433905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:50:42.973226155 (494h21m42.84430919s) [upload=433905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:50:43.270711029 (494h21m43.141794074s) [upload=433905]: Done uploading segments... Dec 26 05:50:43.270744633 (494h21m43.141827668s) [upload=433905]: Upload finished. err: <nil> Duration: 30.002051098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX..X....XXX.XX........XX..X.X............XX...........X..] duration: 30.001996045s trace: https://jaeger.dev.storj.io/trace/6e2bf4b25e4f67b4 FAIL (2025-12-26 05:50:38 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:47:53.117961297 (494h18m52.989044342s) [upload=433866]: Starting upload Dec 26 05:47:53.118081555 (494h18m52.98916459s) [upload=433866]: Done starting upload Dec 26 05:47:53.118124949 (494h18m52.989207984s) [upload=433866]: Uploading segments... Dec 26 05:47:53.118319048 (494h18m52.989402093s) [upload=433866]: Got next segment. Inline: false Dec 26 05:47:53.118387438 (494h18m52.989470483s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:47:53.118394361 (494h18m52.989477406s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:47:53.361835516 (494h18m53.232918561s) [seg_pos={0 0}, upload=433866]: Begin upload segment... Dec 26 05:47:53.361896141 (494h18m53.232979186s) [seg_pos={0 0}, upload=433866]: Done begin upload segment. Dec 26 05:47:53.361944524 (494h18m53.233027569s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.361959562 (494h18m53.233042607s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362685184 (494h18m53.233768219s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362711675 (494h18m53.23379472s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363150640 (494h18m53.234233685s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363591169 (494h18m53.234674214s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364023081 (494h18m53.235106116s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364316040 (494h18m53.235399085s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364965896 (494h18m53.236048942s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365206635 (494h18m53.23628968s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365673104 (494h18m53.236756149s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366176432 (494h18m53.237259467s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366670673 (494h18m53.237753718s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367081676 (494h18m53.238164721s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367454095 (494h18m53.23853714s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.368036294 (494h18m53.239119339s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:55.081468871 (494h18m54.952551906s) [upload=433866]: Next returned nil segment Dec 26 05:47:55.081487868 (494h18m54.952570903s) [upload=433866]: Waiting for error group managing segments... Dec 26 05:47:55.367653326 (494h18m55.238736371s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.368747661 (494h18m55.239830706s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.369426843 (494h18m55.240509889s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.371809802 (494h18m55.242892847s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.382771103 (494h18m55.253854138s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.387164329 (494h18m55.258247364s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.390940750 (494h18m55.262023795s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393284734 (494h18m55.264367769s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393717187 (494h18m55.264800222s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.421978431 (494h18m55.293061466s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.542565630 (494h18m55.413648665s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.609192529 (494h18m55.480275574s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750220822 (494h18m55.621303867s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750241021 (494h18m55.621324056s) [seg_pos={0 0}, upload=433866]: Segment reached optimal threshold of 13 pieces. Dec 26 05:47:55.750532576 (494h18m55.621615621s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750541502 (494h18m55.621624547s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.750884536 (494h18m55.621967571s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750891931 (494h18m55.621974966s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751002631 (494h18m55.622085666s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection Dec 26 05:47:55.751007601 (494h18m55.622090636s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751022178 (494h18m55.622105223s) [seg_pos={0 0}, upload=433866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:47:55.751088264 (494h18m55.622171309s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:47:55.751094225 (494h18m55.62217726s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:47:56.043014072 (494h18m55.914097117s) [upload=433866]: Done uploading segments... Dec 26 05:47:56.043039020 (494h18m55.914122055s) [upload=433866]: Upload finished. err: <nil> Duration: 30.002020952s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..XX..X....XXX.XX........XX..X.X............XX...........X.] duration: 6.736179612s trace: https://jaeger.dev.storj.io/trace/6e2bf4b25e4f67b4 FAIL (2025-12-26 05:49:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:47:53.117961297 (494h18m52.989044342s) [upload=433866]: Starting upload Dec 26 05:47:53.118081555 (494h18m52.98916459s) [upload=433866]: Done starting upload Dec 26 05:47:53.118124949 (494h18m52.989207984s) [upload=433866]: Uploading segments... Dec 26 05:47:53.118319048 (494h18m52.989402093s) [upload=433866]: Got next segment. Inline: false Dec 26 05:47:53.118387438 (494h18m52.989470483s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:47:53.118394361 (494h18m52.989477406s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:47:53.361835516 (494h18m53.232918561s) [seg_pos={0 0}, upload=433866]: Begin upload segment... Dec 26 05:47:53.361896141 (494h18m53.232979186s) [seg_pos={0 0}, upload=433866]: Done begin upload segment. Dec 26 05:47:53.361944524 (494h18m53.233027569s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.361959562 (494h18m53.233042607s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362685184 (494h18m53.233768219s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362711675 (494h18m53.23379472s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363150640 (494h18m53.234233685s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363591169 (494h18m53.234674214s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364023081 (494h18m53.235106116s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364316040 (494h18m53.235399085s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364965896 (494h18m53.236048942s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365206635 (494h18m53.23628968s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365673104 (494h18m53.236756149s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366176432 (494h18m53.237259467s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366670673 (494h18m53.237753718s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367081676 (494h18m53.238164721s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367454095 (494h18m53.23853714s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.368036294 (494h18m53.239119339s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:55.081468871 (494h18m54.952551906s) [upload=433866]: Next returned nil segment Dec 26 05:47:55.081487868 (494h18m54.952570903s) [upload=433866]: Waiting for error group managing segments... Dec 26 05:47:55.367653326 (494h18m55.238736371s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.368747661 (494h18m55.239830706s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.369426843 (494h18m55.240509889s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.371809802 (494h18m55.242892847s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.382771103 (494h18m55.253854138s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.387164329 (494h18m55.258247364s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.390940750 (494h18m55.262023795s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393284734 (494h18m55.264367769s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393717187 (494h18m55.264800222s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.421978431 (494h18m55.293061466s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.542565630 (494h18m55.413648665s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.609192529 (494h18m55.480275574s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750220822 (494h18m55.621303867s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750241021 (494h18m55.621324056s) [seg_pos={0 0}, upload=433866]: Segment reached optimal threshold of 13 pieces. Dec 26 05:47:55.750532576 (494h18m55.621615621s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750541502 (494h18m55.621624547s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.750884536 (494h18m55.621967571s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750891931 (494h18m55.621974966s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751002631 (494h18m55.622085666s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection Dec 26 05:47:55.751007601 (494h18m55.622090636s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751022178 (494h18m55.622105223s) [seg_pos={0 0}, upload=433866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:47:55.751088264 (494h18m55.622171309s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:47:55.751094225 (494h18m55.62217726s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:47:56.043014072 (494h18m55.914097117s) [upload=433866]: Done uploading segments... Dec 26 05:47:56.043039020 (494h18m55.914122055s) [upload=433866]: Upload finished. err: <nil> Duration: 6.73620984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..XX..X....XXX.XX........XX..X.X............XX...........X] duration: 30.002065438s trace: https://jaeger.dev.storj.io/trace/6e2bf4b25e4f67b4 FAIL (2025-12-26 05:48:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 05:47:53.117961297 (494h18m52.989044342s) [upload=433866]: Starting upload Dec 26 05:47:53.118081555 (494h18m52.98916459s) [upload=433866]: Done starting upload Dec 26 05:47:53.118124949 (494h18m52.989207984s) [upload=433866]: Uploading segments... Dec 26 05:47:53.118319048 (494h18m52.989402093s) [upload=433866]: Got next segment. Inline: false Dec 26 05:47:53.118387438 (494h18m52.989470483s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:47:53.118394361 (494h18m52.989477406s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:47:53.361835516 (494h18m53.232918561s) [seg_pos={0 0}, upload=433866]: Begin upload segment... Dec 26 05:47:53.361896141 (494h18m53.232979186s) [seg_pos={0 0}, upload=433866]: Done begin upload segment. Dec 26 05:47:53.361944524 (494h18m53.233027569s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.361959562 (494h18m53.233042607s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362685184 (494h18m53.233768219s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.362711675 (494h18m53.23379472s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363150640 (494h18m53.234233685s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.363591169 (494h18m53.234674214s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364023081 (494h18m53.235106116s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364316040 (494h18m53.235399085s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.364965896 (494h18m53.236048942s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365206635 (494h18m53.23628968s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.365673104 (494h18m53.236756149s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366176432 (494h18m53.237259467s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.366670673 (494h18m53.237753718s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367081676 (494h18m53.238164721s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.367454095 (494h18m53.23853714s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:53.368036294 (494h18m53.239119339s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Uploading piece... Dec 26 05:47:55.081468871 (494h18m54.952551906s) [upload=433866]: Next returned nil segment Dec 26 05:47:55.081487868 (494h18m54.952570903s) [upload=433866]: Waiting for error group managing segments... Dec 26 05:47:55.367653326 (494h18m55.238736371s) [noise=true , address=145.79.227.2:4114 , piece_id=XI2YJNY2RQY5KMYZWWUZVGYAUDYBW2FCFQJWOL4TN7BNMVQ2MKHA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.368747661 (494h18m55.239830706s) [noise=true , address=145.79.227.41:4087 , piece_id=2XX7ZGKYEWXLAUGTVZFAFPGJL7WREA7VS7WKIITN2Y6Y7MTMUDPA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.369426843 (494h18m55.240509889s) [noise=true , address=145.79.227.45:4231 , piece_id=2TXJ7XUGWD5ZSV4DLLNAUHZXBVOPQ2IMD6NBOIBD76CKIZZZGT5Q, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.371809802 (494h18m55.242892847s) [noise=true , address=145.79.227.39:4013 , piece_id=7TNFDSZYASV5P2SMCQTB36TI7SJG5ZVW6A2QIEIXMCZU5CDZ5NKQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.382771103 (494h18m55.253854138s) [noise=true , address=145.79.227.3:4149 , piece_id=QTJJCMN7SREVS3B5RKS63O7CFEQQZHPKUS2RB4PTH45WB4N5US7A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.387164329 (494h18m55.258247364s) [noise=true , address=145.79.227.50:4373 , piece_id=ZVTBDZKQTWW6KURMQRRNN4C54MVBSZ2ZWERLSMCHHUKLZ7IUGAPQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.390940750 (494h18m55.262023795s) [noise=true , address=145.79.227.19:4077 , piece_id=3B6734I3VGXUPIAWXKZ2ANZMY5GSSN5RI7PDIXBM5ZQILDC2TOKA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393284734 (494h18m55.264367769s) [noise=true , address=145.79.227.17:4970 , piece_id=PIQU43P6CLKCQ2YAZGHV6WUEHDWLP6T7JJKJJUQMKTSHI2JB53GA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.393717187 (494h18m55.264800222s) [noise=true , address=145.79.227.38:4977 , piece_id=TZHVCD6LK5EHGPHSJHK5EHVCWEJORVBUI575ZUWCJNQA3YHAUILA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.421978431 (494h18m55.293061466s) [noise=true , address=145.79.227.29:4943 , piece_id=LFXJ5BJ64XGZ3MSOHCRPRUDQDUQJFANB5ABUI7Q5LLLGDIBZLMDA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.542565630 (494h18m55.413648665s) [noise=true , address=77.90.31.17:4770 , piece_id=YZMCZNBQZ3MNBPQX47TB4NWFOQPJHRS7ZSXN7DWPBRWZVDXTHDDQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.609192529 (494h18m55.480275574s) [noise=true , address=77.90.31.10:4517 , piece_id=KY3I57R7A2SCTUBRZRMFMEN26HEEGED6BKC236SMWHITAKDI3K3A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750220822 (494h18m55.621303867s) [noise=true , address=45.91.116.7:4010 , piece_id=37VWEDM3SBFRNTVGSZTHE5DHK3PD7U4MXIKPZ2SDS3ZAMK2UALSQ, seg_pos={0 0}, upload=433866]: Done uploading piece. err: <nil> Dec 26 05:47:55.750241021 (494h18m55.621324056s) [seg_pos={0 0}, upload=433866]: Segment reached optimal threshold of 13 pieces. Dec 26 05:47:55.750532576 (494h18m55.621615621s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750541502 (494h18m55.621624547s) [noise=true , address=45.91.116.8:4048 , piece_id=E2LF6FE7VRRY2G2HAQHK2INBXVLKLSMPMJAU3JQKVVENWHOKTFVA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.750884536 (494h18m55.621967571s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 05:47:55.750891931 (494h18m55.621974966s) [noise=true , address=45.91.116.4:4940 , piece_id=YFJXXITY4QPZOO5AUMMDJWC56ZLG7IEGYU3BIFNHY7R6QHLT5K6A, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751002631 (494h18m55.622085666s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47498->45.91.116.6:4976: use of closed network connection Dec 26 05:47:55.751007601 (494h18m55.622090636s) [noise=true , address=45.91.116.6:4976 , piece_id=R4E5W7LASRA6OZLDE6DWUFOCVABZH6V2ERBGLTCHW3OECC5X5TXA, seg_pos={0 0}, upload=433866]: Long tail cancellation due to optimal threshold Dec 26 05:47:55.751022178 (494h18m55.622105223s) [seg_pos={0 0}, upload=433866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:47:55.751088264 (494h18m55.622171309s) [upload=433866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:47:55.751094225 (494h18m55.62217726s) [upload=433866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:47:56.043014072 (494h18m55.914097117s) [upload=433866]: Done uploading segments... Dec 26 05:47:56.043039020 (494h18m55.914122055s) [upload=433866]: Upload finished. err: <nil> Duration: 30.002090885s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..XX..X....XXX.XX........XX..X.X............XX...........] duration: 6.058403576s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.058428464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X..XX..X....XXX.XX........XX..X.X............XX..........] duration: 13.881591504s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:46:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 13.881619959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..XX..X....XXX.XX........XX..X.X............XX.........] duration: 5.897198855s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:45:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 5.8972277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X....X..XX..X....XXX.XX........XX..X.X............XX........] duration: 5.793356352s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:44:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 5.793384826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X....X..XX..X....XXX.XX........XX..X.X............XX.......] duration: 5.766681376s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 5.766706575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X....X..XX..X....XXX.XX........XX..X.X............XX......] duration: 6.044478557s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:42:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.044514015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X....X..XX..X....XXX.XX........XX..X.X............XX.....] duration: 6.115486797s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.115512477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..X....X..XX..X....XXX.XX........XX..X.X............XX....] duration: 6.007962763s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:41:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.007988713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..X....X..XX..X....XXX.XX........XX..X.X............XX...] duration: 6.171234492s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:39:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.171264568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X..X....X..XX..X....XXX.XX........XX..X.X............XX..] duration: 6.790210897s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:39:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 6.790237807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X..X....X..XX..X....XXX.XX........XX..X.X............XX.] duration: 7.622034339s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 7.622062624s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X..X....X..XX..X....XXX.XX........XX..X.X............XX] duration: 30.003006982s trace: https://jaeger.dev.storj.io/trace/426c33d4da58e3ea FAIL (2025-12-26 05:38:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA) context deadline exceeded ======LOGS====== 69907948s) [upload=433720]: Uploading segments... Dec 26 05:37:46.828141351 (494h8m46.699224386s) [upload=433720]: Got next segment. Inline: false Dec 26 05:37:46.828190294 (494h8m46.699273329s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:46.828195143 (494h8m46.699278188s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:47.073707513 (494h8m46.944790618s) [seg_pos={0 0}, upload=433720]: Begin upload segment... Dec 26 05:37:47.073756846 (494h8m46.944839891s) [seg_pos={0 0}, upload=433720]: Done begin upload segment. Dec 26 05:37:47.073788747 (494h8m46.944871792s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074530369 (494h8m46.945613414s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.074998681 (494h8m46.946081716s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075310365 (494h8m46.9463934s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.075654320 (494h8m46.946737355s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076080772 (494h8m46.947163807s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076366466 (494h8m46.947449501s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.076698208 (494h8m46.947781243s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077062341 (494h8m46.948145376s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.077873977 (494h8m46.948957022s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.078111629 (494h8m46.949194664s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079112535 (494h8m46.95019557s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.079273942 (494h8m46.950356977s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.080264648 (494h8m46.951347693s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.081416051 (494h8m46.952499086s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:47.082736635 (494h8m46.953819671s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Uploading piece... Dec 26 05:37:48.849998594 (494h8m48.721081639s) [upload=433720]: Next returned nil segment Dec 26 05:37:48.850024633 (494h8m48.721107668s) [upload=433720]: Waiting for error group managing segments... Dec 26 05:37:49.246043335 (494h8m49.11712637s) [noise=true , address=145.79.227.42:4121 , piece_id=XDDY7QLR5K5VA4YTAONNQYRONSELD5OHBU4NUK24A2PXQURD6TQQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.248455469 (494h8m49.119538514s) [noise=true , address=145.79.227.26:4655 , piece_id=RCXXBOU7UUG72MLQE7IZXOH6LEGPNEHB25NL6BXWV4Q2LS3DIS2A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.255841653 (494h8m49.126924688s) [noise=true , address=145.79.227.27:4689 , piece_id=5SUQKT72NZEL6ZHZTOHBAEISVFEA2PKYXMR3SDJPABFJEBZTUUCA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.272656211 (494h8m49.143739256s) [noise=true , address=145.79.227.32:4835 , piece_id=RWCRMJJZEPHJN5XK3UC4PWGEIS55DRBNPPWNSY5QPCUWVCM6TT6A, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.282209882 (494h8m49.153292917s) [noise=true , address=145.79.227.43:4158 , piece_id=IXWGLBDHJ472XWW6BFNWOIG3MI7JSNYZJ6N5FEZNY7AEHL2KSDXQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.297128541 (494h8m49.168211576s) [noise=true , address=77.90.31.21:4880 , piece_id=4L4OOHHY3WTKOSUGVZFBLVPTQ36BBXMTFHIQABVEB5VXP3CVY4SA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.301070486 (494h8m49.172153532s) [noise=true , address=45.91.116.6:4976 , piece_id=RWHY53GBYIJDHYQAQ6WV4SBI54K6BTZERX7XFJRHMKWJDRIDHGTQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.313629071 (494h8m49.184712116s) [noise=true , address=145.79.227.41:4087 , piece_id=Q5MOAOCVEVE3J62X3VA4DENEJMRPPFUQ2L2KST65PNEZ2XMAOKOQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.331113515 (494h8m49.20219658s) [noise=true , address=45.91.116.9:4263 , piece_id=HZ5SIWXMFAHHKGJDN22CH3A5X2RSXSHFVWWL5LKSOKA72JAUZ6ZQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.348110429 (494h8m49.219193464s) [noise=true , address=45.91.116.11:4120 , piece_id=A7XDS5VEPULFXJUAAAK5OH4O7VXOG5KRGUO4GJGZEWLOJOB4Z5JA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.375361511 (494h8m49.246444546s) [noise=true , address=45.91.116.12:4226 , piece_id=RPRM47QKTVZPIEFF53APQ5MYODMCAB2FULD6CZMG3EGGQC2KR2MA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.427100511 (494h8m49.298183546s) [noise=true , address=145.79.227.19:4078 , piece_id=DAWL3VJKT6Y2B3TIAKIN6ZDA75WMHKJO4XZRWN6RIOFJC4INGDSQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461899154 (494h8m49.332982189s) [noise=true , address=45.91.116.7:4009 , piece_id=DPXLLRE44XKQLU2DRTF7ORVUIPRUZPAJ64NRHBWYDOJKRF2RZQGA, seg_pos={0 0}, upload=433720]: Done uploading piece. err: <nil> Dec 26 05:37:49.461943068 (494h8m49.333026103s) [seg_pos={0 0}, upload=433720]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:49.462130435 (494h8m49.33321347s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56286->145.79.227.51:4411: use of closed network connection Dec 26 05:37:49.462136696 (494h8m49.333219731s) [noise=true , address=145.79.227.51:4411 , piece_id=KZSKMHMYASCSI6UV2RMTIGBNKJ2YBVZTTLDN236MQ7J4B4DKOB3Q, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462350924 (494h8m49.333433969s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37352->145.79.227.35:4481: use of closed network connection Dec 26 05:37:49.462373027 (494h8m49.333456082s) [noise=true , address=145.79.227.35:4481 , piece_id=7I6RDADIPNPGHSH646L4OL5ZRQIRJRQZBS4LASOCRZPVP4EY2VAQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462396011 (494h8m49.333479106s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:49.462488206 (494h8m49.333571271s) [noise=true , address=45.91.116.4:4937 , piece_id=7G4CQPKWRYUCUNTAIEQLRMLPRLUKTBPWNI6VC72PXQDZZ6HSRHBQ, seg_pos={0 0}, upload=433720]: Long tail cancellation due to optimal threshold Dec 26 05:37:49.462515318 (494h8m49.333598363s) [seg_pos={0 0}, upload=433720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:49.462633824 (494h8m49.333716869s) [upload=433720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:49.462641819 (494h8m49.333724854s) [upload=433720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:49.754951138 (494h8m49.626034183s) [upload=433720]: Done uploading segments... Dec 26 05:37:49.754972329 (494h8m49.626055374s) [upload=433720]: Upload finished. err: <nil> Duration: 30.003035596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.....X..X....X..XX..X....XXX.XX........XX..X.X............X] duration: 30.003227059s trace: https://jaeger.dev.storj.io/trace/42ae78b30ce7fa51 FAIL (2025-12-26 05:37:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: KQUVKDOBCHWGVZTGDXASYWB7MTHMKT64AS7ITQNV62Q3YFM7DLSA) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: SVYHWWGUQWAGGJKITAVEOEWG7BF5BEE6H4KWMQWM243LA6OOCMTQ) context deadline exceeded; piecestore: (Node ID: 12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, Piece ID: CQ5ANM6MHOJUXMWVDXIZHF5IFUN5CQTQ5BNUO7HHWU5B7GZ242RA) context deadline exceeded ======LOGS====== Dec 26 05:37:05.146523348 (494h8m5.017606393s) [upload=433710]: Starting upload Dec 26 05:37:05.146608641 (494h8m5.017691686s) [upload=433710]: Done starting upload Dec 26 05:37:05.146738588 (494h8m5.017821633s) [upload=433710]: Uploading segments... Dec 26 05:37:05.146829171 (494h8m5.017912206s) [upload=433710]: Got next segment. Inline: false Dec 26 05:37:05.146864889 (494h8m5.017947924s) [upload=433710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:37:05.146869477 (494h8m5.017952512s) [upload=433710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:37:05.661283967 (494h8m5.532367012s) [seg_pos={0 0}, upload=433710]: Begin upload segment... Dec 26 05:37:05.661331587 (494h8m5.532414632s) [seg_pos={0 0}, upload=433710]: Done begin upload segment. Dec 26 05:37:05.661360813 (494h8m5.532443858s) [noise=true , address=145.79.227.14:4838 , piece_id=N545MNWVBQHE7EYKYA3CDCL2TDCLAZAUMX2FA2EUD72IQD7Q24HA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.661531638 (494h8m5.532614673s) [noise=true , address=145.79.227.41:4087 , piece_id=O5V4HU3VVHGQL5PLXBSQQSKQMM4W2KIKQ4KHVA23MRGDFFGBZ47Q, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.662118847 (494h8m5.533201882s) [noise=true , address=145.79.227.45:4230 , piece_id=KC3EEECGW2PBUGQ4G3RJNNB5RP45UCYZRF4K3CNZBIOWA7MYKAOA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.662332824 (494h8m5.533415859s) [noise=true , address=145.79.227.3:4150 , piece_id=SXAQLZ4FUR3LY7QZGYDLSHU7R5N2OHCNJLVOCMZCZLWZ6EKSG2YQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.662772260 (494h8m5.533855295s) [noise=true , address=145.79.227.16:4912 , piece_id=KQUVKDOBCHWGVZTGDXASYWB7MTHMKT64AS7ITQNV62Q3YFM7DLSA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.663100396 (494h8m5.534183441s) [noise=true , address=45.91.116.7:4010 , piece_id=2RNRLPHN3MOQZGDVQAUGKVEHANDGHBW5NAC6N44SMY4NM2VNXEZQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.663484717 (494h8m5.534567762s) [noise=true , address=145.79.227.2:4113 , piece_id=KR6HOUNQTJSPAQ7WALAOXHIQWPLMYG5FGLDDLCKHN7FJPT5ARGQA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.663904627 (494h8m5.534987662s) [noise=true , address=145.79.227.11:4732 , piece_id=RWLKT5OTKHU5R37R4SJVF5SOHNT4I62JH7AVGMMK4CUS3YPL4EGQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.664164732 (494h8m5.535247767s) [noise=true , address=45.91.116.6:4976 , piece_id=ZB7XT5YBLZHDLBM6J3D6463I6YJEUATQRR5L3WKO5MTODHEJJJIQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.664719739 (494h8m5.535802774s) [noise=true , address=45.91.116.13:4156 , piece_id=ORK757KCI4UDBQIH4DVDL4Y7DOTS7LZPNQGWQKTMV7FNB7ISDPMQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.664972921 (494h8m5.536055956s) [noise=true , address=45.91.116.5:4298 , piece_id=45OQYHXQBPRRWJP7V33WTGGF6NUQ3QW7PJNV3JGULAYUTSFJQMYQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.665388893 (494h8m5.536471929s) [noise=true , address=145.79.227.51:4410 , piece_id=SVYHWWGUQWAGGJKITAVEOEWG7BF5BEE6H4KWMQWM243LA6OOCMTQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.665769587 (494h8m5.536852623s) [noise=true , address=145.79.227.35:4483 , piece_id=3QQBLX6IJFUGV2MWNOFW5ZMZXEKAVMXDEYHVW3LZYEHAAJD3I4XQ, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.665982814 (494h8m5.537065849s) [noise=true , address=145.79.227.38:4979 , piece_id=C6RMD5QUHRTH7OHCNEPZ7AYO4OBFKNXX7TVC47K3DEJCYG4AVS7A, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.667210211 (494h8m5.538293246s) [noise=true , address=145.79.227.50:4373 , piece_id=OADA6V5VB7D5BBRGP5SBCAFZIJDUWXXCPOKVXJLAWIJTXAGHMQGA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:05.667633607 (494h8m5.538716652s) [noise=true , address=145.79.227.19:4079 , piece_id=CQ5ANM6MHOJUXMWVDXIZHF5IFUN5CQTQ5BNUO7HHWU5B7GZ242RA, seg_pos={0 0}, upload=433710]: Uploading piece... Dec 26 05:37:07.534472808 (494h8m7.405555843s) [upload=433710]: Next returned nil segment Dec 26 05:37:07.534488368 (494h8m7.405571403s) [upload=433710]: Waiting for error group managing segments... Dec 26 05:37:07.836953728 (494h8m7.708036763s) [noise=true , address=145.79.227.38:4979 , piece_id=C6RMD5QUHRTH7OHCNEPZ7AYO4OBFKNXX7TVC47K3DEJCYG4AVS7A, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.841536774 (494h8m7.712619819s) [noise=true , address=145.79.227.50:4373 , piece_id=OADA6V5VB7D5BBRGP5SBCAFZIJDUWXXCPOKVXJLAWIJTXAGHMQGA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.842255964 (494h8m7.713338999s) [noise=true , address=145.79.227.35:4483 , piece_id=3QQBLX6IJFUGV2MWNOFW5ZMZXEKAVMXDEYHVW3LZYEHAAJD3I4XQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.852591014 (494h8m7.723674059s) [noise=true , address=145.79.227.16:4912 , piece_id=KQUVKDOBCHWGVZTGDXASYWB7MTHMKT64AS7ITQNV62Q3YFM7DLSA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.902495280 (494h8m7.773578315s) [noise=true , address=45.91.116.7:4010 , piece_id=2RNRLPHN3MOQZGDVQAUGKVEHANDGHBW5NAC6N44SMY4NM2VNXEZQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:07.915281598 (494h8m7.786364633s) [noise=true , address=145.79.227.14:4838 , piece_id=N545MNWVBQHE7EYKYA3CDCL2TDCLAZAUMX2FA2EUD72IQD7Q24HA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.003739768 (494h8m7.874822813s) [noise=true , address=45.91.116.13:4156 , piece_id=ORK757KCI4UDBQIH4DVDL4Y7DOTS7LZPNQGWQKTMV7FNB7ISDPMQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.553718047 (494h8m8.424801092s) [noise=true , address=145.79.227.19:4079 , piece_id=CQ5ANM6MHOJUXMWVDXIZHF5IFUN5CQTQ5BNUO7HHWU5B7GZ242RA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.564025194 (494h8m8.435108229s) [noise=true , address=145.79.227.51:4410 , piece_id=SVYHWWGUQWAGGJKITAVEOEWG7BF5BEE6H4KWMQWM243LA6OOCMTQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.571642799 (494h8m8.442725834s) [noise=true , address=145.79.227.11:4732 , piece_id=RWLKT5OTKHU5R37R4SJVF5SOHNT4I62JH7AVGMMK4CUS3YPL4EGQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.599604072 (494h8m8.470687107s) [noise=true , address=145.79.227.41:4087 , piece_id=O5V4HU3VVHGQL5PLXBSQQSKQMM4W2KIKQ4KHVA23MRGDFFGBZ47Q, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.629248240 (494h8m8.500331285s) [noise=true , address=45.91.116.6:4976 , piece_id=ZB7XT5YBLZHDLBM6J3D6463I6YJEUATQRR5L3WKO5MTODHEJJJIQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.644664886 (494h8m8.515747921s) [noise=true , address=45.91.116.5:4298 , piece_id=45OQYHXQBPRRWJP7V33WTGGF6NUQ3QW7PJNV3JGULAYUTSFJQMYQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: <nil> Dec 26 05:37:08.644696757 (494h8m8.515779792s) [seg_pos={0 0}, upload=433710]: Segment reached optimal threshold of 13 pieces. Dec 26 05:37:08.644883352 (494h8m8.515966387s) [noise=true , address=145.79.227.45:4230 , piece_id=KC3EEECGW2PBUGQ4G3RJNNB5RP45UCYZRF4K3CNZBIOWA7MYKAOA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis): protocol: write tcp 188.166.225.170:42170->145.79.227.45:4230: use of closed network connection; write tcp 188.166.225.170:42170->145.79.227.45:4230: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42170->145.79.227.45:4230: use of closed network connection Dec 26 05:37:08.644889764 (494h8m8.515972799s) [noise=true , address=145.79.227.45:4230 , piece_id=KC3EEECGW2PBUGQ4G3RJNNB5RP45UCYZRF4K3CNZBIOWA7MYKAOA, seg_pos={0 0}, upload=433710]: Long tail cancellation due to optimal threshold Dec 26 05:37:08.645016776 (494h8m8.516099811s) [noise=true , address=145.79.227.3:4150 , piece_id=SXAQLZ4FUR3LY7QZGYDLSHU7R5N2OHCNJLVOCMZCZLWZ6EKSG2YQ, seg_pos={0 0}, upload=433710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:08.645033818 (494h8m8.516116853s) [noise=true , address=145.79.227.3:4150 , piece_id=SXAQLZ4FUR3LY7QZGYDLSHU7R5N2OHCNJLVOCMZCZLWZ6EKSG2YQ, seg_pos={0 0}, upload=433710]: Long tail cancellation due to optimal threshold Dec 26 05:37:08.645438720 (494h8m8.516521765s) [noise=true , address=145.79.227.2:4113 , piece_id=KR6HOUNQTJSPAQ7WALAOXHIQWPLMYG5FGLDDLCKHN7FJPT5ARGQA, seg_pos={0 0}, upload=433710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: expected piece hash; context canceled; EOF Dec 26 05:37:08.645453327 (494h8m8.516536372s) [noise=true , address=145.79.227.2:4113 , piece_id=KR6HOUNQTJSPAQ7WALAOXHIQWPLMYG5FGLDDLCKHN7FJPT5ARGQA, seg_pos={0 0}, upload=433710]: Long tail cancellation due to optimal threshold Dec 26 05:37:08.645477052 (494h8m8.516560098s) [seg_pos={0 0}, upload=433710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:37:08.645591480 (494h8m8.516674515s) [upload=433710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:37:08.645598474 (494h8m8.516681519s) [upload=433710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:37:08.959316911 (494h8m8.830399956s) [upload=433710]: Done uploading segments... Dec 26 05:37:08.959339103 (494h8m8.830422148s) [upload=433710]: Upload finished. err: <nil> Duration: 30.003249151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.....X..X....X..XX..X....XXX.XX........XX..X.X............] duration: 5.842800857s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.842829071s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.....X..X....X..XX..X....XXX.XX........XX..X.X...........] duration: 18.333355274s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:36:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 18.33338469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XXX.....X..X....X..XX..X....XXX.XX........XX..X.X..........] duration: 5.99417676s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:35:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.994211619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.........] duration: 5.78989822s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.789926874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XXX.....X..X....X..XX..X....XXX.XX........XX..X.X........] duration: 5.831073677s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.831103174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.......] duration: 25.635049205s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:33:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 25.635087248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X......] duration: 6.013638423s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:31:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 6.013666447s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.....] duration: 22.061339044s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:31:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 22.061367307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X....] duration: 15.734063697s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:30:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 15.734096029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X...] duration: 25.610883137s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:30:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 25.610913104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X..] duration: 5.869269758s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:29:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.869294836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X.] duration: 5.983939046s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 5.983966409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.X] duration: 30.002217789s trace: https://jaeger.dev.storj.io/trace/c71d8e314c1b89f FAIL (2025-12-26 05:28:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A) context deadline exceeded ======LOGS====== Dec 26 05:27:52.210375201 (493h58m52.081458236s) [upload=433569]: Starting upload Dec 26 05:27:52.210554512 (493h58m52.081637547s) [upload=433569]: Done starting upload Dec 26 05:27:52.210666595 (493h58m52.08174963s) [upload=433569]: Uploading segments... Dec 26 05:27:52.210759162 (493h58m52.081842207s) [upload=433569]: Got next segment. Inline: false Dec 26 05:27:52.210790952 (493h58m52.081873997s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:52.210795912 (493h58m52.081878957s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:52.459292063 (493h58m52.330375098s) [seg_pos={0 0}, upload=433569]: Begin upload segment... Dec 26 05:27:52.459343491 (493h58m52.330426536s) [seg_pos={0 0}, upload=433569]: Done begin upload segment. Dec 26 05:27:52.459380532 (493h58m52.330463567s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.459525357 (493h58m52.330608392s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460065956 (493h58m52.331149001s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460314150 (493h58m52.331397195s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460622316 (493h58m52.331705351s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.460972093 (493h58m52.332055128s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461310085 (493h58m52.332393131s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461600259 (493h58m52.332683294s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.461910188 (493h58m52.332993223s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462018445 (493h58m52.33310149s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462208466 (493h58m52.333291501s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462478711 (493h58m52.333561756s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.462782680 (493h58m52.333865715s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463101417 (493h58m52.334184452s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.463392992 (493h58m52.334476027s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:52.464428734 (493h58m52.335511769s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Uploading piece... Dec 26 05:27:53.640059950 (493h58m53.511142995s) [noise=true , address=77.90.31.21:4877 , piece_id=6RMO3YGAX2BTYP35AARFA6ZHQQNWH2FHZEVRHVITNLOQVQ5CT4YA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 26 05:27:54.079381578 (493h58m53.950464613s) [upload=433569]: Next returned nil segment Dec 26 05:27:54.079394623 (493h58m53.950477668s) [upload=433569]: Waiting for error group managing segments... Dec 26 05:27:54.462914624 (493h58m54.333997669s) [noise=true , address=145.79.227.49:4339 , piece_id=IEXUEKA3XCQMB3HNAXUQ45HMZRZEOKCMCXT46773VOHYIDJXKJCA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.465155863 (493h58m54.336238898s) [noise=true , address=145.79.227.7:4293 , piece_id=FD6XKSALLCFLTDEHCMWI76MPA6JAREFE73EERWBKWHCIZBC2BWGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.466619680 (493h58m54.337702715s) [noise=true , address=145.79.227.15:4875 , piece_id=WRV676RDSGYHG7OTE6B3G6C6TRDRUE5NPE2ZU4TCP6Y7EPM3OICA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.467645784 (493h58m54.33872884s) [noise=true , address=145.79.227.20:4437 , piece_id=6PL2CUXWNZJKV2A7LUOOUOOZ66SLEKUQJ2N76FNDMLBIVGX7Z76Q, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.470507453 (493h58m54.341590488s) [noise=true , address=145.79.227.41:4086 , piece_id=3XNP3QZ3RHPOXTTJTE6A7MHSA5OE3VMKQTDSD7LZIQ5HUYOZ5CPA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.471648236 (493h58m54.342731281s) [noise=true , address=145.79.227.36:4517 , piece_id=66YHYE3SQKUZNSBDBLRRIFBITZVD5D5IT7ST23Q43XGVFVS3RIVQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.483800676 (493h58m54.354883711s) [noise=true , address=145.79.227.42:4123 , piece_id=DEAMVHAKL37MFAM3VHZ765QN3XPB5NE336LEZJCZSWOKDDCIOOOA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.489180930 (493h58m54.360263965s) [noise=true , address=145.79.227.51:4411 , piece_id=WR2MKKIELB3FPHPQDWBGZZD2SGJTELRUXCZJAGYHSXFIRJVBZD7A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.494996395 (493h58m54.36607946s) [noise=true , address=145.79.227.32:4835 , piece_id=NZKGN7EGVQ7MYYVL4JZMPJYKPMBPOXTV4VIC56WQCU3YL5LJMJXQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.659337770 (493h58m54.530420805s) [noise=true , address=77.90.31.15:4700 , piece_id=VPUT5CLZ7V3YVKIO7EIDBVOAHEFUEKMHAKEG4QB7Z7S6QSBWTNJA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.705960088 (493h58m54.577043133s) [noise=true , address=45.91.116.5:4298 , piece_id=ZVWJEVOOHQ6J5RIV7L6XY3L2MG2GELQ7VP7542VMFNLS2C7GRF5A, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.712229125 (493h58m54.58331217s) [noise=true , address=77.90.31.10:4518 , piece_id=AWXSB6JBPILLXBCKYYSEKEMXETXZMC6A3QFUPEW6PZISBX5SUJHA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714895782 (493h58m54.585978827s) [noise=true , address=45.91.116.11:4119 , piece_id=OEYIHCIA6MB3YFXT2YXQ7JDYP3XI42DDTT23T6NQZPI7EULNRPHQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: <nil> Dec 26 05:27:54.714910852 (493h58m54.585993887s) [seg_pos={0 0}, upload=433569]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:54.715086847 (493h58m54.586169892s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42124->77.90.31.13:4629: use of closed network connection Dec 26 05:27:54.715092577 (493h58m54.586175613s) [noise=true , address=77.90.31.13:4629 , piece_id=KLLEAUWV4XDXOGQCULXTGURLGWEXAQJQGAR3FFUEIKOH7EBDMKFA, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715429209 (493h58m54.586512264s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:54.715439037 (493h58m54.586522072s) [noise=true , address=45.91.116.7:4011 , piece_id=Q57ESMSCDN2LBKBM47QSTDJKFHASKPZWUWMKAADFYTMY6N77NDGQ, seg_pos={0 0}, upload=433569]: Long tail cancellation due to optimal threshold Dec 26 05:27:54.715464606 (493h58m54.586547651s) [seg_pos={0 0}, upload=433569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:54.715546933 (493h58m54.586629968s) [upload=433569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:54.715553375 (493h58m54.58663642s) [upload=433569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:55.029380258 (493h58m54.900463303s) [upload=433569]: Done uploading segments... Dec 26 05:27:55.029399986 (493h58m54.900483021s) [upload=433569]: Upload finished. err: <nil> Duration: 30.002237697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X.] duration: 6.520618834s trace: https://jaeger.dev.storj.io/trace/7c0fe85218674364 FAIL (2025-12-26 05:27:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h58m5.645567836s) [upload=433555]: Uploading segments... Dec 26 05:27:05.774715801 (493h58m5.645798916s) [upload=433555]: Got next segment. Inline: false Dec 26 05:27:05.774765486 (493h58m5.645848531s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:05.774772640 (493h58m5.645855685s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:06.016795314 (493h58m5.887878359s) [seg_pos={0 0}, upload=433555]: Begin upload segment... Dec 26 05:27:06.016858164 (493h58m5.887941209s) [seg_pos={0 0}, upload=433555]: Done begin upload segment. Dec 26 05:27:06.016915854 (493h58m5.887998899s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.016936362 (493h58m5.888019408s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017483865 (493h58m5.88856691s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017919154 (493h58m5.889002189s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.018363831 (493h58m5.889446876s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019015862 (493h58m5.890098898s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019442785 (493h58m5.89052582s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019923531 (493h58m5.891006566s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.020919908 (493h58m5.892002953s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.021335268 (493h58m5.892418313s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022267123 (493h58m5.893350168s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022956917 (493h58m5.894039962s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.024694646 (493h58m5.895777691s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.026549077 (493h58m5.897632112s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.028447342 (493h58m5.899530387s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.032267697 (493h58m5.903350732s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:07.650527267 (493h58m7.521610302s) [upload=433555]: Next returned nil segment Dec 26 05:27:07.650584757 (493h58m7.521667832s) [upload=433555]: Waiting for error group managing segments... Dec 26 05:27:08.033475289 (493h58m7.904558334s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035352323 (493h58m7.906435368s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035995929 (493h58m7.907078974s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.037031842 (493h58m7.908114887s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.050209093 (493h58m7.921292138s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.058686837 (493h58m7.929769882s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.063734358 (493h58m7.934817403s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.285416371 (493h58m8.156499416s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.315376310 (493h58m8.186459355s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350677299 (493h58m8.221760344s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350934860 (493h58m8.222017915s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.358680749 (493h58m8.229763794s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364120138 (493h58m8.235203173s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364144233 (493h58m8.235227268s) [seg_pos={0 0}, upload=433555]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:08.364378429 (493h58m8.235461464s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection Dec 26 05:27:08.364391353 (493h58m8.235474388s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364487386 (493h58m8.235570431s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection Dec 26 05:27:08.364497055 (493h58m8.23558009s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364512384 (493h58m8.235595419s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:08.364518125 (493h58m8.23560116s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364536400 (493h58m8.235619435s) [seg_pos={0 0}, upload=433555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:08.364608057 (493h58m8.235691102s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:08.364614048 (493h58m8.235697113s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:08.650304575 (493h58m8.52138761s) [upload=433555]: Done uploading segments... Dec 26 05:27:08.650324582 (493h58m8.521407618s) [upload=433555]: Upload finished. err: <nil> Duration: 6.520634575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..X] duration: 30.009340972s trace: https://jaeger.dev.storj.io/trace/7c0fe85218674364 FAIL (2025-12-26 05:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 493h58m5.645567836s) [upload=433555]: Uploading segments... Dec 26 05:27:05.774715801 (493h58m5.645798916s) [upload=433555]: Got next segment. Inline: false Dec 26 05:27:05.774765486 (493h58m5.645848531s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:27:05.774772640 (493h58m5.645855685s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:27:06.016795314 (493h58m5.887878359s) [seg_pos={0 0}, upload=433555]: Begin upload segment... Dec 26 05:27:06.016858164 (493h58m5.887941209s) [seg_pos={0 0}, upload=433555]: Done begin upload segment. Dec 26 05:27:06.016915854 (493h58m5.887998899s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.016936362 (493h58m5.888019408s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017483865 (493h58m5.88856691s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.017919154 (493h58m5.889002189s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.018363831 (493h58m5.889446876s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019015862 (493h58m5.890098898s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019442785 (493h58m5.89052582s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.019923531 (493h58m5.891006566s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.020919908 (493h58m5.892002953s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.021335268 (493h58m5.892418313s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022267123 (493h58m5.893350168s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.022956917 (493h58m5.894039962s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.024694646 (493h58m5.895777691s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.026549077 (493h58m5.897632112s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.028447342 (493h58m5.899530387s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:06.032267697 (493h58m5.903350732s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Uploading piece... Dec 26 05:27:07.650527267 (493h58m7.521610302s) [upload=433555]: Next returned nil segment Dec 26 05:27:07.650584757 (493h58m7.521667832s) [upload=433555]: Waiting for error group managing segments... Dec 26 05:27:08.033475289 (493h58m7.904558334s) [noise=true , address=145.79.227.51:4409 , piece_id=JVG3EDSJGG4E6IVHFP3DDVTTUQUXIRZ5IU6RLTLV57ANUA32IDNA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035352323 (493h58m7.906435368s) [noise=true , address=145.79.227.29:4941 , piece_id=R7LTOZVQDNYXHXAINHWWR323XBZV5M2G5O4FSOGAKFJFEOPMXQRA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.035995929 (493h58m7.907078974s) [noise=true , address=145.79.227.16:4912 , piece_id=KNM3A3FMWPIG3N4F3CF6U5NSFFZ5PWH4T6NBR4YL7IU63ETPABKQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.037031842 (493h58m7.908114887s) [noise=true , address=145.79.227.14:4837 , piece_id=4ABGCEX4PHXWP46TREYRNXP46MP5LKXI2WPSAGD4SAULXPEDPITA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.050209093 (493h58m7.921292138s) [noise=true , address=145.79.227.35:4483 , piece_id=3ZKMUUVFQ5BHTPNGWXU3H3HM5CCDAFSZZN24GAIM73STLC3GVNIA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.058686837 (493h58m7.929769882s) [noise=true , address=145.79.227.15:4873 , piece_id=JTFFCII5D6JO3H2H4YJF3LFD6CYRL4U47ALOL5MB7Z6LKSVTRWWA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.063734358 (493h58m7.934817403s) [noise=true , address=145.79.227.18:4007 , piece_id=LHKT2LVCXNFDSYZ4JNC7UDKAVZ4O5FDXS3AGO7UJRAP3D77CKYFA, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.285416371 (493h58m8.156499416s) [noise=true , address=45.91.116.12:4226 , piece_id=6FYKJOZZM5KITZ6BJPYFJANEIFYI6XD2GV67G5WLGDBTNS4YKTMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.315376310 (493h58m8.186459355s) [noise=true , address=77.90.31.13:4629 , piece_id=DATUCRALJR525IKFMH5HWMXBBVXFY63M7QWP47D5QHZEDMAF3SDQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350677299 (493h58m8.221760344s) [noise=true , address=45.91.116.14:4192 , piece_id=E3A75BLQK6PWCF6JVGV2P4CLRBJILP2PNWSJMXKEOKEGDJYOWVIQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.350934860 (493h58m8.222017915s) [noise=true , address=77.90.31.10:4520 , piece_id=WKYV2ZU3TLPCEBVOF5CDGLVOKGVABMBUU6XJTGRAP66FZTAK7B2A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.358680749 (493h58m8.229763794s) [noise=true , address=77.90.31.19:4844 , piece_id=4FWQS4RLTJZHQTP6GO6Y2Y5D4NEBNL2HESLONVTH2KWROTBSTTJQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364120138 (493h58m8.235203173s) [noise=true , address=45.91.116.13:4154 , piece_id=QSZEJ4EKS75WUTGTTXFYQNJILE2NB62BUZ2HQYSDIJVES32DPHPQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: <nil> Dec 26 05:27:08.364144233 (493h58m8.235227268s) [seg_pos={0 0}, upload=433555]: Segment reached optimal threshold of 13 pieces. Dec 26 05:27:08.364378429 (493h58m8.235461464s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33208->45.91.116.4:4937: use of closed network connection Dec 26 05:27:08.364391353 (493h58m8.235474388s) [noise=true , address=45.91.116.4:4937 , piece_id=HORSAGSEVJATQNVNIXIVCEONAAJAXBXWJF44XPHHJIW5TJBZ633A, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364487386 (493h58m8.235570431s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41028->77.90.31.14:4664: use of closed network connection Dec 26 05:27:08.364497055 (493h58m8.23558009s) [noise=true , address=77.90.31.14:4664 , piece_id=XNJCATTAECI3HKFMY5PAHJ7YS5AVOXCI4BBYYBLRN3BZGVMFYT4Q, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364512384 (493h58m8.235595419s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 05:27:08.364518125 (493h58m8.23560116s) [noise=true , address=45.91.116.8:4048 , piece_id=PJVNYRCY3QC6QU7HRED43YB7GM5GOUFTPCIV36C5JSMWLTUHKAMQ, seg_pos={0 0}, upload=433555]: Long tail cancellation due to optimal threshold Dec 26 05:27:08.364536400 (493h58m8.235619435s) [seg_pos={0 0}, upload=433555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:27:08.364608057 (493h58m8.235691102s) [upload=433555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:27:08.364614048 (493h58m8.235697113s) [upload=433555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:27:08.650304575 (493h58m8.52138761s) [upload=433555]: Done uploading segments... Dec 26 05:27:08.650324582 (493h58m8.521407618s) [upload=433555]: Upload finished. err: <nil> Duration: 30.009364858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX..] duration: 5.789046939s trace: https://jaeger.dev.storj.io/trace/59744209fa1c6ce8 FAIL (2025-12-26 05:26:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA) context deadline exceeded ======LOGS====== 2 (493h56m3.727143957s) [upload=433529]: Uploading segments... Dec 26 05:25:03.856315957 (493h56m3.727399002s) [upload=433529]: Got next segment. Inline: false Dec 26 05:25:03.856369630 (493h56m3.727452675s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:25:03.856376773 (493h56m3.727459818s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:25:04.111517796 (493h56m3.982600842s) [seg_pos={0 0}, upload=433529]: Begin upload segment... Dec 26 05:25:04.111583542 (493h56m3.982666597s) [seg_pos={0 0}, upload=433529]: Done begin upload segment. Dec 26 05:25:04.111625051 (493h56m3.982708106s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112072593 (493h56m3.983155628s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112473336 (493h56m3.983556381s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113099338 (493h56m3.984182383s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113476937 (493h56m3.984559972s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114075596 (493h56m3.985158642s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114364087 (493h56m3.985447142s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114626285 (493h56m3.98570933s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115009866 (493h56m3.986092911s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115065272 (493h56m3.986148317s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115447590 (493h56m3.986530635s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115471935 (493h56m3.98655498s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115946439 (493h56m3.987029484s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116437795 (493h56m3.98752084s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116981520 (493h56m3.988064575s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.117478346 (493h56m3.988561381s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:05.732596514 (493h56m5.603679549s) [upload=433529]: Next returned nil segment Dec 26 05:25:05.732623395 (493h56m5.60370643s) [upload=433529]: Waiting for error group managing segments... Dec 26 05:25:06.117422330 (493h56m5.988505375s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121075256 (493h56m5.992158311s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121835634 (493h56m5.992918679s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.151720319 (493h56m6.022803364s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.153245173 (493h56m6.024328218s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.374541640 (493h56m6.245624685s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.386806795 (493h56m6.25788983s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.406499814 (493h56m6.277582859s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.419187393 (493h56m6.290270438s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425344617 (493h56m6.296427652s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425867934 (493h56m6.296950979s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.526235201 (493h56m6.397318246s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590590335 (493h56m6.46167338s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590613259 (493h56m6.461696304s) [seg_pos={0 0}, upload=433529]: Segment reached optimal threshold of 13 pieces. Dec 26 05:25:06.590830773 (493h56m6.461913818s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection Dec 26 05:25:06.590836694 (493h56m6.461919739s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.590961211 (493h56m6.462044246s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:25:06.590973354 (493h56m6.462056389s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591072112 (493h56m6.462155147s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection Dec 26 05:25:06.591081209 (493h56m6.462164244s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591138178 (493h56m6.462221223s) [seg_pos={0 0}, upload=433529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:25:06.591238730 (493h56m6.462321765s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:25:06.591246915 (493h56m6.46232995s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:25:06.887867373 (493h56m6.758950418s) [upload=433529]: Done uploading segments... Dec 26 05:25:06.887944439 (493h56m6.759027504s) [upload=433529]: Upload finished. err: <nil> Duration: 5.789065565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX.] duration: 22.195178349s trace: https://jaeger.dev.storj.io/trace/59744209fa1c6ce8 FAIL (2025-12-26 05:26:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA) context deadline exceeded ======LOGS====== 2 (493h56m3.727143957s) [upload=433529]: Uploading segments... Dec 26 05:25:03.856315957 (493h56m3.727399002s) [upload=433529]: Got next segment. Inline: false Dec 26 05:25:03.856369630 (493h56m3.727452675s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:25:03.856376773 (493h56m3.727459818s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:25:04.111517796 (493h56m3.982600842s) [seg_pos={0 0}, upload=433529]: Begin upload segment... Dec 26 05:25:04.111583542 (493h56m3.982666597s) [seg_pos={0 0}, upload=433529]: Done begin upload segment. Dec 26 05:25:04.111625051 (493h56m3.982708106s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112072593 (493h56m3.983155628s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112473336 (493h56m3.983556381s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113099338 (493h56m3.984182383s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113476937 (493h56m3.984559972s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114075596 (493h56m3.985158642s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114364087 (493h56m3.985447142s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114626285 (493h56m3.98570933s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115009866 (493h56m3.986092911s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115065272 (493h56m3.986148317s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115447590 (493h56m3.986530635s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115471935 (493h56m3.98655498s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115946439 (493h56m3.987029484s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116437795 (493h56m3.98752084s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116981520 (493h56m3.988064575s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.117478346 (493h56m3.988561381s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:05.732596514 (493h56m5.603679549s) [upload=433529]: Next returned nil segment Dec 26 05:25:05.732623395 (493h56m5.60370643s) [upload=433529]: Waiting for error group managing segments... Dec 26 05:25:06.117422330 (493h56m5.988505375s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121075256 (493h56m5.992158311s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121835634 (493h56m5.992918679s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.151720319 (493h56m6.022803364s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.153245173 (493h56m6.024328218s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.374541640 (493h56m6.245624685s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.386806795 (493h56m6.25788983s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.406499814 (493h56m6.277582859s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.419187393 (493h56m6.290270438s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425344617 (493h56m6.296427652s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425867934 (493h56m6.296950979s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.526235201 (493h56m6.397318246s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590590335 (493h56m6.46167338s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590613259 (493h56m6.461696304s) [seg_pos={0 0}, upload=433529]: Segment reached optimal threshold of 13 pieces. Dec 26 05:25:06.590830773 (493h56m6.461913818s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection Dec 26 05:25:06.590836694 (493h56m6.461919739s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.590961211 (493h56m6.462044246s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:25:06.590973354 (493h56m6.462056389s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591072112 (493h56m6.462155147s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection Dec 26 05:25:06.591081209 (493h56m6.462164244s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591138178 (493h56m6.462221223s) [seg_pos={0 0}, upload=433529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:25:06.591238730 (493h56m6.462321765s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:25:06.591246915 (493h56m6.46232995s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:25:06.887867373 (493h56m6.758950418s) [upload=433529]: Done uploading segments... Dec 26 05:25:06.887944439 (493h56m6.759027504s) [upload=433529]: Upload finished. err: <nil> Duration: 22.195212132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.....X.....XXX.....X..X....X..XX..X....XXX.XX........XX] duration: 30.002828049s trace: https://jaeger.dev.storj.io/trace/59744209fa1c6ce8 FAIL (2025-12-26 05:25:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA) context deadline exceeded ======LOGS====== 2 (493h56m3.727143957s) [upload=433529]: Uploading segments... Dec 26 05:25:03.856315957 (493h56m3.727399002s) [upload=433529]: Got next segment. Inline: false Dec 26 05:25:03.856369630 (493h56m3.727452675s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:25:03.856376773 (493h56m3.727459818s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:25:04.111517796 (493h56m3.982600842s) [seg_pos={0 0}, upload=433529]: Begin upload segment... Dec 26 05:25:04.111583542 (493h56m3.982666597s) [seg_pos={0 0}, upload=433529]: Done begin upload segment. Dec 26 05:25:04.111625051 (493h56m3.982708106s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112072593 (493h56m3.983155628s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.112473336 (493h56m3.983556381s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113099338 (493h56m3.984182383s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.113476937 (493h56m3.984559972s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114075596 (493h56m3.985158642s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114364087 (493h56m3.985447142s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.114626285 (493h56m3.98570933s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115009866 (493h56m3.986092911s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115065272 (493h56m3.986148317s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115447590 (493h56m3.986530635s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115471935 (493h56m3.98655498s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.115946439 (493h56m3.987029484s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116437795 (493h56m3.98752084s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.116981520 (493h56m3.988064575s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:04.117478346 (493h56m3.988561381s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Uploading piece... Dec 26 05:25:05.732596514 (493h56m5.603679549s) [upload=433529]: Next returned nil segment Dec 26 05:25:05.732623395 (493h56m5.60370643s) [upload=433529]: Waiting for error group managing segments... Dec 26 05:25:06.117422330 (493h56m5.988505375s) [noise=true , address=145.79.227.29:4941 , piece_id=ZEHET6U5VYZ32JGALG5PTBLAQ7BMTEM4NR36MIE7GZV5KUBTLHMA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121075256 (493h56m5.992158311s) [noise=true , address=145.79.227.51:4409 , piece_id=XVECS57O6LAMRQMQ6STRDAY4BK4CC3SDUXSPEZW4HT3WMWJKJZGQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.121835634 (493h56m5.992918679s) [noise=true , address=145.79.227.50:4374 , piece_id=OD7EXRPKUBEZBIQ377NKTX2ANH3KVCGI4RIF5BYHTHOESEVJCYOA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.151720319 (493h56m6.022803364s) [noise=true , address=145.79.227.52:4446 , piece_id=3IRG2YZC375GSXGRYCHH7PBJ4G57L75QKWHADN7FURYJRNL2VI4A, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.153245173 (493h56m6.024328218s) [noise=true , address=145.79.227.16:4912 , piece_id=M7SLSPFVNBU5JA2GEMS4ZG64XR4I5SHJPO4E3CZ7DBOZXYK5SXZQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.374541640 (493h56m6.245624685s) [noise=true , address=45.91.116.4:4939 , piece_id=GY5JZS6TGV4QRAUR7LLILZUUWYCWZAYF6VBGCRFQKEQ76PEAU3PQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.386806795 (493h56m6.25788983s) [noise=true , address=77.90.31.11:4554 , piece_id=DYSYHTBVSCBDH7LKOVAXPLZQ427JB6MANT7NNKERMC3SDWMH25MA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.406499814 (493h56m6.277582859s) [noise=true , address=77.90.31.15:4700 , piece_id=B6ASMO3FT22NAXUJNCVG2APMNNFF3DHMNTPQSJEYN7D63HHQGYHQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.419187393 (493h56m6.290270438s) [noise=true , address=77.90.31.14:4664 , piece_id=KQSXRIFAUOAEBYMIEPQBPJ2VVGITMJ267XBOZ436FAKHTGKU2UCQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425344617 (493h56m6.296427652s) [noise=true , address=77.90.31.10:4519 , piece_id=2HNNR7T4X4SC5XSIPRF2FF2G2SMTQJJRFMY3C2AXZ3SUWBTIFLJA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.425867934 (493h56m6.296950979s) [noise=true , address=45.91.116.8:4047 , piece_id=KANCDIUR6DQWZJETCYP6RQUH7BAF437KPK32JLBNQJSBO4OPTZSA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.526235201 (493h56m6.397318246s) [noise=true , address=77.90.31.17:4770 , piece_id=J2DPGKK7SM2IEW4RCBURCIAXWMD4PRKMGAXOP2OBOTJYTC332VEA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590590335 (493h56m6.46167338s) [noise=true , address=45.91.116.7:4010 , piece_id=NFIL4DI44ZKGOJC4UVXAM3FWIHJ6DQF2KQWH2K4Y4VMWJJWOV4PA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: <nil> Dec 26 05:25:06.590613259 (493h56m6.461696304s) [seg_pos={0 0}, upload=433529]: Segment reached optimal threshold of 13 pieces. Dec 26 05:25:06.590830773 (493h56m6.461913818s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57542->77.90.31.13:4627: use of closed network connection Dec 26 05:25:06.590836694 (493h56m6.461919739s) [noise=true , address=77.90.31.13:4627 , piece_id=KV26POUAKWG3UDCIRAOG34ADIDEXKJX6OUDJT5B3YSP2B4IFKGZA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.590961211 (493h56m6.462044246s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 05:25:06.590973354 (493h56m6.462056389s) [noise=true , address=145.79.227.44:4194 , piece_id=MX2D3EHLWP6POTP5NFGZRG6X7FXXUBMXW3HVSU4SPG25FQYW2BKQ, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591072112 (493h56m6.462155147s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49286->45.91.116.9:4262: use of closed network connection Dec 26 05:25:06.591081209 (493h56m6.462164244s) [noise=true , address=45.91.116.9:4262 , piece_id=QQCR735FITBAT7KM6KG5F5XYGBKQY4TLRDDC5PXDIOCXXPTKCGAA, seg_pos={0 0}, upload=433529]: Long tail cancellation due to optimal threshold Dec 26 05:25:06.591138178 (493h56m6.462221223s) [seg_pos={0 0}, upload=433529]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:25:06.591238730 (493h56m6.462321765s) [upload=433529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:25:06.591246915 (493h56m6.46232995s) [upload=433529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:25:06.887867373 (493h56m6.758950418s) [upload=433529]: Done uploading segments... Dec 26 05:25:06.887944439 (493h56m6.759027504s) [upload=433529]: Upload finished. err: <nil> Duration: 30.002961072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X.....X.....XXX.....X..X....X..XX..X....XXX.XX........X] duration: 30.002931627s trace: https://jaeger.dev.storj.io/trace/4bfb7a10a5e66e37 FAIL (2025-12-26 05:24:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 05:24:30.635265506 (493h55m30.506348551s) [upload=433524]: Starting upload Dec 26 05:24:30.635323726 (493h55m30.506406761s) [upload=433524]: Done starting upload Dec 26 05:24:30.635501535 (493h55m30.50658458s) [upload=433524]: Uploading segments... Dec 26 05:24:30.635622976 (493h55m30.506706021s) [upload=433524]: Got next segment. Inline: false Dec 26 05:24:30.635686898 (493h55m30.506769943s) [upload=433524]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:24:30.635694773 (493h55m30.506777818s) [upload=433524]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:24:30.878490688 (493h55m30.749573723s) [seg_pos={0 0}, upload=433524]: Begin upload segment... Dec 26 05:24:30.878549781 (493h55m30.749632826s) [seg_pos={0 0}, upload=433524]: Done begin upload segment. Dec 26 05:24:30.878581792 (493h55m30.749664827s) [noise=true , address=77.90.31.13:4627 , piece_id=X4SEM6MDLPZ7DITDJB5J5I3WWVLRLB4UJOWQDJOS3K6IT6LTLOHQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.879474052 (493h55m30.750557097s) [noise=true , address=145.79.227.49:4338 , piece_id=JSGYDEJKSU6JF5CKJA6W2F7LTDQX7FZTSKEGMOZ6UNAQRJBKPODQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.879929769 (493h55m30.751012804s) [noise=true , address=145.79.227.17:4970 , piece_id=PQUKBWMUZC2V553DDBXRX4XBKI7RVO72RUJMOSIGC4J4K5PO455A, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.880340721 (493h55m30.751423766s) [noise=true , address=145.79.227.47:4266 , piece_id=V2XLASYHAYL3MPOEVNNH2CQODHJZL2Z4PZ2CKCPKJL34PAIHMIWQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.880810095 (493h55m30.75189314s) [noise=true , address=145.79.227.40:4049 , piece_id=67E52BC5VXNFL5PAHRR76HMI2EPEUK2UWYV7YWJCFL2ULJ2PBLHA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.881106289 (493h55m30.752189334s) [noise=true , address=145.79.227.44:4193 , piece_id=6YNRHKXY6ADEYVPNMYIFSGW4XYLHTGTZMZALCECLU26BJP7WIGGQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.881452548 (493h55m30.752535593s) [noise=true , address=77.90.31.15:4700 , piece_id=7BMYJO4VKXHGEGGZ7TVYSI5NQ5XLMDWDJA6K57APR4BEPZFLFDHA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.882909252 (493h55m30.753992297s) [noise=true , address=145.79.227.6:4257 , piece_id=GDN6H2JKHSR36ODZAZM3PZITQAASOCEQ2CWBRNLEQIDERBUJ7VHA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.884191013 (493h55m30.755274058s) [noise=true , address=77.90.31.10:4520 , piece_id=FYFSUFWCS3T7G3SI4OEIF7CK2SGFASTQMOMN5PXOW4LFBQUGQ5EQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.885479466 (493h55m30.756562501s) [noise=true , address=145.79.227.35:4483 , piece_id=DL4IWXX6ABS4UNN5VLMRIZWAQBEMNA6VNSGRULYJHXVU7OZMWK7Q, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.887080695 (493h55m30.75816374s) [noise=true , address=145.79.227.14:4837 , piece_id=JOHT4FK7G3AGDCKNUH5GUY34KM2ZGTDB3PPFMA2G37I4SXSVIWSQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.889862152 (493h55m30.760945197s) [noise=true , address=145.79.227.11:4732 , piece_id=VXECZGKDPARDZTEA4RAM4V64JMCA2NSYC2ZURTSYQRQX6YA5S62Q, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.893917604 (493h55m30.765000649s) [noise=true , address=145.79.227.29:4942 , piece_id=VB2A6UJ46AG353ATY2DDOK62CSSW2BBPO26GFVUT5TFB4JXMDVAQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.896934779 (493h55m30.768017834s) [noise=true , address=77.90.31.17:4770 , piece_id=JN37L7NJIOZR6JV4MPRNSFZTVDYJ72K5VAYOVJY4OKAJBTA6APPA, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.900705750 (493h55m30.771788785s) [noise=true , address=145.79.227.48:4301 , piece_id=4TZSO26266XINC6ADPYSY5TOIUVS4Q4VO25YMC5XPV3XUBWKA5QQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:30.900862398 (493h55m30.771945433s) [noise=true , address=145.79.227.34:4905 , piece_id=5FNKEV7FO2VG4YXOD52NJN4EBVQXQFFIEWXN2HQ4YQDDGATXEDAQ, seg_pos={0 0}, upload=433524]: Uploading piece... Dec 26 05:24:32.597903994 (493h55m32.468987029s) [upload=433524]: Next returned nil segment Dec 26 05:24:32.597912189 (493h55m32.468995224s) [upload=433524]: Waiting for error group managing segments... Dec 26 05:24:32.891572500 (493h55m32.762655535s) [noise=true , address=145.79.227.29:4942 , piece_id=VB2A6UJ46AG353ATY2DDOK62CSSW2BBPO26GFVUT5TFB4JXMDVAQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.892716649 (493h55m32.763799684s) [noise=true , address=145.79.227.6:4257 , piece_id=GDN6H2JKHSR36ODZAZM3PZITQAASOCEQ2CWBRNLEQIDERBUJ7VHA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.899428870 (493h55m32.770511916s) [noise=true , address=145.79.227.17:4970 , piece_id=PQUKBWMUZC2V553DDBXRX4XBKI7RVO72RUJMOSIGC4J4K5PO455A, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.903887140 (493h55m32.774970175s) [noise=true , address=145.79.227.48:4301 , piece_id=4TZSO26266XINC6ADPYSY5TOIUVS4Q4VO25YMC5XPV3XUBWKA5QQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.907788749 (493h55m32.778871784s) [noise=true , address=145.79.227.49:4338 , piece_id=JSGYDEJKSU6JF5CKJA6W2F7LTDQX7FZTSKEGMOZ6UNAQRJBKPODQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.918161260 (493h55m32.789244295s) [noise=true , address=145.79.227.11:4732 , piece_id=VXECZGKDPARDZTEA4RAM4V64JMCA2NSYC2ZURTSYQRQX6YA5S62Q, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.920643717 (493h55m32.791726762s) [noise=true , address=145.79.227.35:4483 , piece_id=DL4IWXX6ABS4UNN5VLMRIZWAQBEMNA6VNSGRULYJHXVU7OZMWK7Q, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.947528541 (493h55m32.818611576s) [noise=true , address=145.79.227.47:4266 , piece_id=V2XLASYHAYL3MPOEVNNH2CQODHJZL2Z4PZ2CKCPKJL34PAIHMIWQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:32.953434246 (493h55m32.824517291s) [noise=true , address=145.79.227.14:4837 , piece_id=JOHT4FK7G3AGDCKNUH5GUY34KM2ZGTDB3PPFMA2G37I4SXSVIWSQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.077551427 (493h55m32.948634462s) [noise=true , address=77.90.31.13:4627 , piece_id=X4SEM6MDLPZ7DITDJB5J5I3WWVLRLB4UJOWQDJOS3K6IT6LTLOHQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.080225649 (493h55m32.951308694s) [noise=true , address=77.90.31.15:4700 , piece_id=7BMYJO4VKXHGEGGZ7TVYSI5NQ5XLMDWDJA6K57APR4BEPZFLFDHA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.112798524 (493h55m32.983881569s) [noise=true , address=145.79.227.34:4905 , piece_id=5FNKEV7FO2VG4YXOD52NJN4EBVQXQFFIEWXN2HQ4YQDDGATXEDAQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.113458821 (493h55m32.984541856s) [noise=true , address=77.90.31.10:4520 , piece_id=FYFSUFWCS3T7G3SI4OEIF7CK2SGFASTQMOMN5PXOW4LFBQUGQ5EQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: <nil> Dec 26 05:24:33.113478668 (493h55m32.984561713s) [seg_pos={0 0}, upload=433524]: Segment reached optimal threshold of 13 pieces. Dec 26 05:24:33.113763340 (493h55m32.984846375s) [noise=true , address=145.79.227.44:4193 , piece_id=6YNRHKXY6ADEYVPNMYIFSGW4XYLHTGTZMZALCECLU26BJP7WIGGQ, seg_pos={0 0}, upload=433524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 05:24:33.113770014 (493h55m32.984853059s) [noise=true , address=145.79.227.44:4193 , piece_id=6YNRHKXY6ADEYVPNMYIFSGW4XYLHTGTZMZALCECLU26BJP7WIGGQ, seg_pos={0 0}, upload=433524]: Long tail cancellation due to optimal threshold Dec 26 05:24:33.113869904 (493h55m32.984952949s) [noise=true , address=145.79.227.40:4049 , piece_id=67E52BC5VXNFL5PAHRR76HMI2EPEUK2UWYV7YWJCFL2ULJ2PBLHA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 05:24:33.113882077 (493h55m32.984965122s) [noise=true , address=145.79.227.40:4049 , piece_id=67E52BC5VXNFL5PAHRR76HMI2EPEUK2UWYV7YWJCFL2ULJ2PBLHA, seg_pos={0 0}, upload=433524]: Long tail cancellation due to optimal threshold Dec 26 05:24:33.114040398 (493h55m32.985123443s) [noise=true , address=77.90.31.17:4770 , piece_id=JN37L7NJIOZR6JV4MPRNSFZTVDYJ72K5VAYOVJY4OKAJBTA6APPA, seg_pos={0 0}, upload=433524]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 26 05:24:33.114049105 (493h55m32.98513215s) [noise=true , address=77.90.31.17:4770 , piece_id=JN37L7NJIOZR6JV4MPRNSFZTVDYJ72K5VAYOVJY4OKAJBTA6APPA, seg_pos={0 0}, upload=433524]: Long tail cancellation due to optimal threshold Dec 26 05:24:33.114069734 (493h55m32.985152769s) [seg_pos={0 0}, upload=433524]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:24:33.114157862 (493h55m32.985240907s) [upload=433524]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:24:33.114163212 (493h55m32.985246257s) [upload=433524]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:24:33.416191437 (493h55m33.287274472s) [upload=433524]: Done uploading segments... Dec 26 05:24:33.416211656 (493h55m33.287294701s) [upload=433524]: Upload finished. err: <nil> Duration: 30.00295457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.....X.....XXX.....X..X....X..XX..X....XXX.XX........] duration: 5.799094765s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 5.799116516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX.......] duration: 23.932184439s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:22:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 23.932212361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX......] duration: 6.022888017s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:22:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.022914527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX.....] duration: 6.108342368s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.108371383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X......X.....X.....XXX.....X..X....X..XX..X....XXX.XX....] duration: 6.038089854s trace: https://jaeger.dev.storj.io/trace/4be771a9ae73237c FAIL (2025-12-26 05:21:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6 05:19:40.015831336 (493h50m39.886914371s) [upload=433448]: Starting upload Dec 26 05:19:40.015901219 (493h50m39.886984255s) [upload=433448]: Done starting upload Dec 26 05:19:40.016005167 (493h50m39.887088212s) [upload=433448]: Uploading segments... Dec 26 05:19:40.016187505 (493h50m39.88727055s) [upload=433448]: Got next segment. Inline: false Dec 26 05:19:40.016234224 (493h50m39.887317269s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 05:19:40.016239875 (493h50m39.88732292s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 05:19:40.259498121 (493h50m40.130581156s) [seg_pos={0 0}, upload=433448]: Begin upload segment... Dec 26 05:19:40.259552846 (493h50m40.130635881s) [seg_pos={0 0}, upload=433448]: Done begin upload segment. Dec 26 05:19:40.259582793 (493h50m40.130665838s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.259589476 (493h50m40.130672521s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260172425 (493h50m40.13125547s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.260578338 (493h50m40.131661373s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261572802 (493h50m40.132655837s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.261998523 (493h50m40.133081568s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.262456414 (493h50m40.133539449s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263634869 (493h50m40.134717904s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.263680094 (493h50m40.134763129s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.265092354 (493h50m40.136175399s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.267205177 (493h50m40.138288212s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.269251734 (493h50m40.140334779s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.270498768 (493h50m40.141581813s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.271739471 (493h50m40.142822506s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.272062416 (493h50m40.143145461s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:40.273449487 (493h50m40.144532532s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Uploading piece... Dec 26 05:19:41.974165209 (493h50m41.845248254s) [upload=433448]: Next returned nil segment Dec 26 05:19:41.974183082 (493h50m41.845266117s) [upload=433448]: Waiting for error group managing segments... Dec 26 05:19:42.250888097 (493h50m42.121971132s) [noise=true , address=145.79.227.13:4804 , piece_id=IP52ICQUEH7MJ2VEN2X2YEGYR3ADHYLXHET6MU24JITIZWSPQQOQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.275447373 (493h50m42.146530408s) [noise=true , address=145.79.227.17:4971 , piece_id=7Y5RYVNGLNS3TJHJIFXRR3BOUY7FX2CX37CSDGZ7PDWA426P75YQ, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.277929178 (493h50m42.149012214s) [noise=true , address=145.79.227.4:4186 , piece_id=XZZCYSG7TJX2CTXEEPYUYE5QFKE7TNYGHPVVXZ5QT4K7JRZ2FYMA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.282127542 (493h50m42.153210597s) [noise=true , address=145.79.227.50:4375 , piece_id=O67BEQULXVN3EA4TDZSW5WPD4HLLATXV4VF623BNL7KOE3HTDB6Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.286103714 (493h50m42.157186759s) [noise=true , address=145.79.227.29:4941 , piece_id=DI3P3L2LK2KG2YLB32I5PLTXN4QQD6K56ISVBC3EZJ33WMKNP3TA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.288008270 (493h50m42.159091305s) [noise=true , address=145.79.227.6:4257 , piece_id=3ISMQOQEYYAKN4HS7ZLZBK72O6LB52MNBKTVN7BYNXQCHAGLPMRA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.306718969 (493h50m42.177802014s) [noise=true , address=145.79.227.47:4266 , piece_id=CYKRMGA5WKZZMWK3DXT7UWPKDJFQ6YHONP42C565EUPVUVJNLJIA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.310090630 (493h50m42.181173675s) [noise=true , address=145.79.227.52:4446 , piece_id=R5HDBPETX7OX42XCJT2WZGLV7GV5XEA75GVLK7WAIMCYX27CUIDA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.490004708 (493h50m42.361087753s) [noise=true , address=77.90.31.11:4554 , piece_id=425GGDABUPUZ3XDOCXF5FDSWXYTUV54KTBJ537DGJUHUESJNZ2HA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.528943815 (493h50m42.40002685s) [noise=true , address=45.91.116.14:4191 , piece_id=RTMNWED3YDDMNSM4FXT5UIE2MJW4RMK46Y2YEVZEXKT56PXDI2LA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.586233994 (493h50m42.45731704s) [noise=true , address=77.90.31.18:4808 , piece_id=RXEXYCQHPBAXG26DZP4575BULW2VQCN2CGOFCJSDEAPVORBGPH2A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.587509283 (493h50m42.458592318s) [noise=true , address=145.79.227.33:4871 , piece_id=5D23WIXBKSVQGRSRWT7PIJSW3QDX43LNDDKQSLTJMC6EM656TPFA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595262287 (493h50m42.466345322s) [noise=true , address=45.91.116.13:4153 , piece_id=ZNVO2N7C5SXFEMMLXYTVKJ6MZGXQFOH57ZWXFMZVGIWCQ45NCH4Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: <nil> Dec 26 05:19:42.595284690 (493h50m42.466367735s) [seg_pos={0 0}, upload=433448]: Segment reached optimal threshold of 13 pieces. Dec 26 05:19:42.595464903 (493h50m42.466547938s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595471505 (493h50m42.46655455s) [noise=true , address=45.91.116.12:4225 , piece_id=VLDTTCIK33YAYUWRI3RYXB4WKZANOMY6QV5IV2LYJJYU3S4JG3JA, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595619286 (493h50m42.466702331s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 26 05:19:42.595643112 (493h50m42.466726157s) [noise=true , address=45.91.116.6:4976 , piece_id=3RAODBZJM4NCVGDIUV2ZTH726KUAX7DUQRVZYFBTZMWJXL6IO76A, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595662899 (493h50m42.466745934s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49752->45.91.116.11:4120: use of closed network connection Dec 26 05:19:42.595669402 (493h50m42.466752437s) [noise=true , address=45.91.116.11:4120 , piece_id=YTELBPZN5CUD55CGSQXAD2QAJPURAC4ICHNUSWYYJ3ST33TME45Q, seg_pos={0 0}, upload=433448]: Long tail cancellation due to optimal threshold Dec 26 05:19:42.595767758 (493h50m42.466850794s) [seg_pos={0 0}, upload=433448]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 05:19:42.595843854 (493h50m42.466926889s) [upload=433448]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 05:19:42.595848493 (493h50m42.466931528s) [upload=433448]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 05:19:42.881439589 (493h50m42.752522634s) [upload=433448]: Done uploading segments... Dec 26 05:19:42.881463124 (493h50m42.752546159s) [upload=433448]: Upload finished. err: <nil> Duration: 6.038115201s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)