Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X....XX] duration: 30.00340172s trace: https://jaeger.dev.storj.io/trace/8fa2249477c2a2e FAIL (2026-01-02 01:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 3YA55UITNRTSMMOVBYCTHZWRCAVBLZX3ZVPUXIXWSX2LXGOPX52A) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: KPMSRFQSFY43FZZOKD2ABLJI2TDCAAESQOABY26O3MIWWFKNHFVQ) context deadline exceeded ======LOGS====== 01:21:42.070637895 (657h52m41.94172093s) [upload=576326]: Starting upload Jan 2 01:21:42.070770527 (657h52m41.941853562s) [upload=576326]: Done starting upload Jan 2 01:21:42.070874656 (657h52m41.941957701s) [upload=576326]: Uploading segments... Jan 2 01:21:42.071225353 (657h52m41.942308449s) [upload=576326]: Got next segment. Inline: false Jan 2 01:21:42.071281460 (657h52m41.942364505s) [upload=576326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:21:42.071288202 (657h52m41.942371247s) [upload=576326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:21:42.316839111 (657h52m42.187922146s) [seg_pos={0 0}, upload=576326]: Begin upload segment... Jan 2 01:21:42.316892022 (657h52m42.187975067s) [seg_pos={0 0}, upload=576326]: Done begin upload segment. Jan 2 01:21:42.316919975 (657h52m42.18800301s) [noise=true , address=145.79.227.17:4972 , piece_id=LHR5NBXLFU5LNPGDVNXRRKTUZM7QWHPEQ7DLA5UEIZPU4ABOXYXA, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.317208114 (657h52m42.188291159s) [noise=true , address=145.79.227.12:4767 , piece_id=NWEJJON2WGB4MAAKNWY56TLKRZV3RZDUVJQ4TM3HL7MTFJLT5ULQ, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.318282730 (657h52m42.189365765s) [noise=true , address=145.79.227.50:4375 , piece_id=NJ3VTGTR7KZ2WNJPM7QGYXOYA4O6JK3XT2B7M27KZRZVXPWKW57A, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.318718960 (657h52m42.189802005s) [noise=true , address=145.79.227.11:4729 , piece_id=QTBV6NF6HT7UAJTY7ZFK6UNXMAAGUDKQKI7NVENI4GPD4XQWCHRA, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.319083635 (657h52m42.19016667s) [noise=true , address=145.79.227.7:4294 , piece_id=535IQQEKS6WHJEVC3OLODIVBH7Q5VVBIPWS66YHHEDTAC7RPZTBA, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.319469730 (657h52m42.190552765s) [noise=true , address=145.79.227.28:4725 , piece_id=ZITJV5XWBRZZSZ5FQDCVSUQMHMN5WTHTBMYXQM25BJERGFZBG7KQ, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.319780352 (657h52m42.190863397s) [noise=true , address=145.79.227.25:4618 , piece_id=LBVFQPX4ZYPGSLHCNZ5C7QO66BDC4G7CFQYNUL2M2HHSWL35AHSA, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.321079325 (657h52m42.19216236s) [noise=true , address=145.79.227.38:4977 , piece_id=W4HIMRSADQ7ENXKQQ7VNZKDD25F7YJHCHYKPAFLFET3FH3MCG2VA, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.321249098 (657h52m42.192332144s) [noise=true , address=45.91.116.12:4228 , piece_id=L46VI3ORYAEVIKJDZTZCADAQGSIUHZKPFE5W4ZP7HCA3CHJPY2MA, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.322452600 (657h52m42.193535645s) [noise=true , address=45.91.116.5:4299 , piece_id=KPMSRFQSFY43FZZOKD2ABLJI2TDCAAESQOABY26O3MIWWFKNHFVQ, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.324302834 (657h52m42.195385869s) [noise=true , address=145.79.227.29:4942 , piece_id=IPSV5WB477SPBZJIF6NEXGVMJAFWMQBS5E7EHVFSYAQQIKXTRWWQ, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.325864687 (657h52m42.196947732s) [noise=true , address=45.91.116.7:4012 , piece_id=XSO4DFYS32HEHNSV26NN2AEEZVX45DIDDSLHAK3EKUBZDNL3WY3A, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.327298066 (657h52m42.198381101s) [noise=true , address=145.79.227.23:4546 , piece_id=OLBXIZVOXCPLCGFSCZOG6H7RVTVOYT7OJLFC36YOCZLMGKCJ2S7Q, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.328409302 (657h52m42.199492337s) [noise=true , address=145.79.227.45:4229 , piece_id=SSLESUXD4TVZVWLGOSTUEGQOH6T5FNWREKVP2TSOIWQXMOUGYB2A, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.329722533 (657h52m42.200805578s) [noise=true , address=145.79.227.52:4446 , piece_id=ZZXLR3O5JADIA3TORYMNCTZSYP2D7V734FKHCESZP7F4DZ5BB4JQ, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:42.331142307 (657h52m42.202225372s) [noise=true , address=77.90.31.12:4590 , piece_id=3YA55UITNRTSMMOVBYCTHZWRCAVBLZX3ZVPUXIXWSX2LXGOPX52A, seg_pos={0 0}, upload=576326]: Uploading piece... Jan 2 01:21:44.029512889 (657h52m43.900595924s) [upload=576326]: Next returned nil segment Jan 2 01:21:44.029526314 (657h52m43.900609349s) [upload=576326]: Waiting for error group managing segments... Jan 2 01:21:44.304715568 (657h52m44.175798613s) [noise=true , address=145.79.227.11:4729 , piece_id=QTBV6NF6HT7UAJTY7ZFK6UNXMAAGUDKQKI7NVENI4GPD4XQWCHRA, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.309779590 (657h52m44.180862645s) [noise=true , address=145.79.227.50:4375 , piece_id=NJ3VTGTR7KZ2WNJPM7QGYXOYA4O6JK3XT2B7M27KZRZVXPWKW57A, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.322422514 (657h52m44.193505559s) [noise=true , address=145.79.227.25:4618 , piece_id=LBVFQPX4ZYPGSLHCNZ5C7QO66BDC4G7CFQYNUL2M2HHSWL35AHSA, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.322903430 (657h52m44.193986475s) [noise=true , address=145.79.227.45:4229 , piece_id=SSLESUXD4TVZVWLGOSTUEGQOH6T5FNWREKVP2TSOIWQXMOUGYB2A, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.327695995 (657h52m44.19877903s) [noise=true , address=145.79.227.38:4977 , piece_id=W4HIMRSADQ7ENXKQQ7VNZKDD25F7YJHCHYKPAFLFET3FH3MCG2VA, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.327721304 (657h52m44.198804339s) [noise=true , address=145.79.227.52:4446 , piece_id=ZZXLR3O5JADIA3TORYMNCTZSYP2D7V734FKHCESZP7F4DZ5BB4JQ, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.335945003 (657h52m44.207028038s) [noise=true , address=145.79.227.29:4942 , piece_id=IPSV5WB477SPBZJIF6NEXGVMJAFWMQBS5E7EHVFSYAQQIKXTRWWQ, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.336416730 (657h52m44.207499776s) [noise=true , address=145.79.227.28:4725 , piece_id=ZITJV5XWBRZZSZ5FQDCVSUQMHMN5WTHTBMYXQM25BJERGFZBG7KQ, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.336782127 (657h52m44.207865162s) [noise=true , address=145.79.227.7:4294 , piece_id=535IQQEKS6WHJEVC3OLODIVBH7Q5VVBIPWS66YHHEDTAC7RPZTBA, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.337458484 (657h52m44.208541529s) [noise=true , address=145.79.227.12:4767 , piece_id=NWEJJON2WGB4MAAKNWY56TLKRZV3RZDUVJQ4TM3HL7MTFJLT5ULQ, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.406424945 (657h52m44.27750799s) [noise=true , address=145.79.227.17:4972 , piece_id=LHR5NBXLFU5LNPGDVNXRRKTUZM7QWHPEQ7DLA5UEIZPU4ABOXYXA, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.699441694 (657h52m44.570524739s) [noise=true , address=77.90.31.12:4590 , piece_id=3YA55UITNRTSMMOVBYCTHZWRCAVBLZX3ZVPUXIXWSX2LXGOPX52A, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.739370634 (657h52m44.610453679s) [noise=true , address=45.91.116.5:4299 , piece_id=KPMSRFQSFY43FZZOKD2ABLJI2TDCAAESQOABY26O3MIWWFKNHFVQ, seg_pos={0 0}, upload=576326]: Done uploading piece. err: <nil> Jan 2 01:21:44.739391413 (657h52m44.610474448s) [seg_pos={0 0}, upload=576326]: Segment reached optimal threshold of 13 pieces. Jan 2 01:21:44.739572197 (657h52m44.610655232s) [noise=true , address=45.91.116.12:4228 , piece_id=L46VI3ORYAEVIKJDZTZCADAQGSIUHZKPFE5W4ZP7HCA3CHJPY2MA, seg_pos={0 0}, upload=576326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Jan 2 01:21:44.739579131 (657h52m44.610662166s) [noise=true , address=45.91.116.12:4228 , piece_id=L46VI3ORYAEVIKJDZTZCADAQGSIUHZKPFE5W4ZP7HCA3CHJPY2MA, seg_pos={0 0}, upload=576326]: Long tail cancellation due to optimal threshold Jan 2 01:21:44.739688880 (657h52m44.610771915s) [noise=true , address=45.91.116.7:4012 , piece_id=XSO4DFYS32HEHNSV26NN2AEEZVX45DIDDSLHAK3EKUBZDNL3WY3A, seg_pos={0 0}, upload=576326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Jan 2 01:21:44.739693679 (657h52m44.610776714s) [noise=true , address=45.91.116.7:4012 , piece_id=XSO4DFYS32HEHNSV26NN2AEEZVX45DIDDSLHAK3EKUBZDNL3WY3A, seg_pos={0 0}, upload=576326]: Long tail cancellation due to optimal threshold Jan 2 01:21:44.739789521 (657h52m44.610872566s) [noise=true , address=145.79.227.23:4546 , piece_id=OLBXIZVOXCPLCGFSCZOG6H7RVTVOYT7OJLFC36YOCZLMGKCJ2S7Q, seg_pos={0 0}, upload=576326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:42656->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:42656->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42656->145.79.227.23:4546: use of closed network connection Jan 2 01:21:44.739822234 (657h52m44.610905269s) [noise=true , address=145.79.227.23:4546 , piece_id=OLBXIZVOXCPLCGFSCZOG6H7RVTVOYT7OJLFC36YOCZLMGKCJ2S7Q, seg_pos={0 0}, upload=576326]: Long tail cancellation due to optimal threshold Jan 2 01:21:44.739870866 (657h52m44.610953901s) [seg_pos={0 0}, upload=576326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:21:44.740004721 (657h52m44.611087756s) [upload=576326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:21:44.740016202 (657h52m44.611099238s) [upload=576326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:21:45.039051542 (657h52m44.910134577s) [upload=576326]: Done uploading segments... Jan 2 01:21:45.039073443 (657h52m44.910156478s) [upload=576326]: Upload finished. err: <nil> Duration: 30.003571613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 33.645896043s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:21:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 33.645922674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.511364514s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:21:52 UTC) orbiter warning: context deadline exceeded Duration: 3.511392286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.278547751s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:21:44 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.278586715s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X....X] duration: 30.002557832s trace: https://jaeger.dev.storj.io/trace/68da54a2ff56f28a FAIL (2026-01-02 01:21:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: 42CZ3F2RBERWJSPI4DPZY2QAP3WWY6EEA2LT4GZT35IKMXKYJMDA) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HG2LI7NCRXXJQV2LZAENWWNXC5AYH5UMN5UI5YJUTS4FKBYXFYMA) context deadline exceeded ======LOGS====== Jan 2 01:21:10.707374205 (657h52m10.57845725s) [upload=576314]: Starting upload Jan 2 01:21:10.707469908 (657h52m10.578552953s) [upload=576314]: Done starting upload Jan 2 01:21:10.707522067 (657h52m10.578605112s) [upload=576314]: Uploading segments... Jan 2 01:21:10.707731245 (657h52m10.57881429s) [upload=576314]: Got next segment. Inline: false Jan 2 01:21:10.707783305 (657h52m10.57886636s) [upload=576314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:21:10.707789586 (657h52m10.578872631s) [upload=576314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:21:10.984272705 (657h52m10.85535575s) [seg_pos={0 0}, upload=576314]: Begin upload segment... Jan 2 01:21:10.984345723 (657h52m10.855428759s) [seg_pos={0 0}, upload=576314]: Done begin upload segment. Jan 2 01:21:10.984389076 (657h52m10.855472111s) [noise=true , address=145.79.227.40:4049 , piece_id=MA3QCW52J7ZBIWYWBEMMZSZCR54CVZFKPOYRIUZHP5EKT4K3HXWQ, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.984421858 (657h52m10.855504893s) [noise=true , address=45.91.116.7:4010 , piece_id=CG2ITVNP736NYIO65T64GQEZHPABHZEGJASNSRLIEWI2J6BOR55A, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.985340328 (657h52m10.856423363s) [noise=true , address=77.90.31.17:4773 , piece_id=42CZ3F2RBERWJSPI4DPZY2QAP3WWY6EEA2LT4GZT35IKMXKYJMDA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.985689793 (657h52m10.856772838s) [noise=true , address=145.79.227.13:4803 , piece_id=XACKOK3K2CUQ4CW2OMK4PIY2GJ5RZ5AWUUEI4QV6ICFGAJ7SJCGQ, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.986150500 (657h52m10.857233545s) [noise=true , address=145.79.227.31:4798 , piece_id=T32WL4OLGCNKNCEEPTCT5RZ343JLRHL3KN4IU2IR6MJHHMTUTS2A, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.986495728 (657h52m10.857578763s) [noise=true , address=145.79.227.29:4941 , piece_id=I63HQDAVDY2YIHEAJBWOQDBAIHOEEPADMOTOYPFVHROL4JT5HXZA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.986800238 (657h52m10.857883273s) [noise=true , address=45.91.116.10:4083 , piece_id=27LKSHQZZC2XI7XVJAJAAVL7JWKUCQFGXAYE75R5UVLBQREAZWEQ, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.987138471 (657h52m10.858221506s) [noise=true , address=145.79.227.5:4223 , piece_id=4DI776BSDRDF6SBQ7OQHKPBTHYQMJTQDII7K24HZEDNBTFZZVNZA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.987506172 (657h52m10.858589217s) [noise=true , address=145.79.227.42:4121 , piece_id=D5SXIDE6PO62IXV4QQ6GL63BGGGYWGEVIJZTWPRKAHCGOVK3VH6A, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.987883811 (657h52m10.858966856s) [noise=true , address=45.91.116.13:4154 , piece_id=HG2LI7NCRXXJQV2LZAENWWNXC5AYH5UMN5UI5YJUTS4FKBYXFYMA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.988142603 (657h52m10.859225638s) [noise=true , address=145.79.227.41:4087 , piece_id=IA2FC2OR3C5U7NTTIAE3YMEIC6ZSIL73SMMUCDJVDNYV6OR7BN6Q, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.988533127 (657h52m10.859616172s) [noise=true , address=145.79.227.28:4725 , piece_id=4RW62SNAUFQ3KVF2WLSTU7RZGDUE3G2A5PIRR3ZHZXVUKEB6FBNA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.990015520 (657h52m10.861098555s) [noise=true , address=145.79.227.10:4403 , piece_id=PJSTGGF5SHQW4DSD7E4NWSPIRIAZP7R6L7O5537NLBUZBOGFVWHA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.990611074 (657h52m10.861694119s) [noise=true , address=45.91.116.14:4189 , piece_id=KH7Z7ITX5N623AYZAFR2XOM7WLSJQ74O34Y66TPLOXOD5MCBR4SQ, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.991139009 (657h52m10.862222044s) [noise=true , address=145.79.227.30:4761 , piece_id=35DRZSQZMHPTWA27ALNY6S5PUPLN5IQMSSZR63DMHLBV4NIMKMHA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:10.992563421 (657h52m10.863646466s) [noise=true , address=145.79.227.33:4870 , piece_id=2KY2QOBJYXDGRGQPOZHD4LZHRKCVOLG4IDHKVJP7TZGGHXZHNBRA, seg_pos={0 0}, upload=576314]: Uploading piece... Jan 2 01:21:12.702596992 (657h52m12.573680027s) [upload=576314]: Next returned nil segment Jan 2 01:21:12.702611559 (657h52m12.573694594s) [upload=576314]: Waiting for error group managing segments... Jan 2 01:21:12.974071532 (657h52m12.845154577s) [noise=true , address=145.79.227.13:4803 , piece_id=XACKOK3K2CUQ4CW2OMK4PIY2GJ5RZ5AWUUEI4QV6ICFGAJ7SJCGQ, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.009047801 (657h52m12.880130836s) [noise=true , address=145.79.227.5:4223 , piece_id=4DI776BSDRDF6SBQ7OQHKPBTHYQMJTQDII7K24HZEDNBTFZZVNZA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.011596193 (657h52m12.882679238s) [noise=true , address=145.79.227.41:4087 , piece_id=IA2FC2OR3C5U7NTTIAE3YMEIC6ZSIL73SMMUCDJVDNYV6OR7BN6Q, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.019555188 (657h52m12.890638223s) [noise=true , address=145.79.227.28:4725 , piece_id=4RW62SNAUFQ3KVF2WLSTU7RZGDUE3G2A5PIRR3ZHZXVUKEB6FBNA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.070277670 (657h52m12.941360705s) [noise=true , address=145.79.227.10:4403 , piece_id=PJSTGGF5SHQW4DSD7E4NWSPIRIAZP7R6L7O5537NLBUZBOGFVWHA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.084204218 (657h52m12.955287263s) [noise=true , address=145.79.227.31:4798 , piece_id=T32WL4OLGCNKNCEEPTCT5RZ343JLRHL3KN4IU2IR6MJHHMTUTS2A, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.198577378 (657h52m13.069660413s) [noise=true , address=77.90.31.17:4773 , piece_id=42CZ3F2RBERWJSPI4DPZY2QAP3WWY6EEA2LT4GZT35IKMXKYJMDA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.445901113 (657h52m13.316984148s) [noise=true , address=45.91.116.14:4189 , piece_id=KH7Z7ITX5N623AYZAFR2XOM7WLSJQ74O34Y66TPLOXOD5MCBR4SQ, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.629525256 (657h52m13.500608291s) [noise=true , address=45.91.116.7:4010 , piece_id=CG2ITVNP736NYIO65T64GQEZHPABHZEGJASNSRLIEWI2J6BOR55A, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:13.776358624 (657h52m13.647441669s) [noise=true , address=45.91.116.10:4083 , piece_id=27LKSHQZZC2XI7XVJAJAAVL7JWKUCQFGXAYE75R5UVLBQREAZWEQ, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:14.567634749 (657h52m14.438717784s) [noise=true , address=145.79.227.33:4870 , piece_id=2KY2QOBJYXDGRGQPOZHD4LZHRKCVOLG4IDHKVJP7TZGGHXZHNBRA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:14.597302250 (657h52m14.468385285s) [noise=true , address=45.91.116.13:4154 , piece_id=HG2LI7NCRXXJQV2LZAENWWNXC5AYH5UMN5UI5YJUTS4FKBYXFYMA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:14.892992159 (657h52m14.764075204s) [noise=true , address=145.79.227.40:4049 , piece_id=MA3QCW52J7ZBIWYWBEMMZSZCR54CVZFKPOYRIUZHP5EKT4K3HXWQ, seg_pos={0 0}, upload=576314]: Done uploading piece. err: <nil> Jan 2 01:21:14.893013009 (657h52m14.764096054s) [seg_pos={0 0}, upload=576314]: Segment reached optimal threshold of 13 pieces. Jan 2 01:21:14.893214381 (657h52m14.764297426s) [noise=true , address=145.79.227.30:4761 , piece_id=35DRZSQZMHPTWA27ALNY6S5PUPLN5IQMSSZR63DMHLBV4NIMKMHA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Jan 2 01:21:14.893224511 (657h52m14.764307556s) [noise=true , address=145.79.227.30:4761 , piece_id=35DRZSQZMHPTWA27ALNY6S5PUPLN5IQMSSZR63DMHLBV4NIMKMHA, seg_pos={0 0}, upload=576314]: Long tail cancellation due to optimal threshold Jan 2 01:21:14.893439200 (657h52m14.764522245s) [noise=true , address=145.79.227.29:4941 , piece_id=I63HQDAVDY2YIHEAJBWOQDBAIHOEEPADMOTOYPFVHROL4JT5HXZA, seg_pos={0 0}, upload=576314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Jan 2 01:21:14.893465290 (657h52m14.764548335s) [noise=true , address=145.79.227.29:4941 , piece_id=I63HQDAVDY2YIHEAJBWOQDBAIHOEEPADMOTOYPFVHROL4JT5HXZA, seg_pos={0 0}, upload=576314]: Long tail cancellation due to optimal threshold Jan 2 01:21:14.893515195 (657h52m14.76459824s) [noise=true , address=145.79.227.42:4121 , piece_id=D5SXIDE6PO62IXV4QQ6GL63BGGGYWGEVIJZTWPRKAHCGOVK3VH6A, seg_pos={0 0}, upload=576314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Jan 2 01:21:14.893525143 (657h52m14.764608188s) [noise=true , address=145.79.227.42:4121 , piece_id=D5SXIDE6PO62IXV4QQ6GL63BGGGYWGEVIJZTWPRKAHCGOVK3VH6A, seg_pos={0 0}, upload=576314]: Long tail cancellation due to optimal threshold Jan 2 01:21:14.893633890 (657h52m14.764716925s) [seg_pos={0 0}, upload=576314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:21:14.893789477 (657h52m14.764872532s) [upload=576314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:21:14.893827139 (657h52m14.764910184s) [upload=576314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:21:15.188647512 (657h52m15.059730547s) [upload=576314]: Done uploading segments... Jan 2 01:21:15.188667080 (657h52m15.059750125s) [upload=576314]: Upload finished. err: <nil> Duration: 30.002584853s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 151.920866ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:21:40 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 151.961293ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.422243164s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:21:26 UTC) orbiter warning: context deadline exceeded Duration: 3.422268482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 192.636464ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:21:07 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 192.663505ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 51.216379163s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 51.216407186s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.633471928s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:20:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.63348937s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 197.278492ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:20:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 197.306266ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.688756841s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:20:31 UTC) orbiter warning: context deadline exceeded Duration: 3.688783501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 142.463959ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:20:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 142.500737ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X....] duration: 6.209834531s trace: https://jaeger.dev.storj.io/trace/5d8db3215b5f0048 FAIL (2026-01-02 01:20:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:17:27.018977618 (657h48m26.890060663s) [upload=576263]: Starting upload Jan 2 01:17:27.019050036 (657h48m26.890133101s) [upload=576263]: Done starting upload Jan 2 01:17:27.019151209 (657h48m26.890234244s) [upload=576263]: Uploading segments... Jan 2 01:17:27.019322414 (657h48m26.89040546s) [upload=576263]: Got next segment. Inline: false Jan 2 01:17:27.019366899 (657h48m26.890449945s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:17:27.019371849 (657h48m26.890454894s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:17:27.262407008 (657h48m27.133490043s) [seg_pos={0 0}, upload=576263]: Begin upload segment... Jan 2 01:17:27.262468044 (657h48m27.133551089s) [seg_pos={0 0}, upload=576263]: Done begin upload segment. Jan 2 01:17:27.262501878 (657h48m27.133584913s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.262578323 (657h48m27.133661368s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263142959 (657h48m27.134225994s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263550565 (657h48m27.1346336s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263937091 (657h48m27.135020126s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264322504 (657h48m27.13540554s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264717267 (657h48m27.135800312s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265237397 (657h48m27.136320442s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265458929 (657h48m27.136541964s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266082807 (657h48m27.137165872s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266599510 (657h48m27.137682545s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.269500053 (657h48m27.140583088s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.270195437 (657h48m27.141278482s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.271042040 (657h48m27.142125085s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.272118508 (657h48m27.143201544s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.273731761 (657h48m27.144814806s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:28.874153517 (657h48m28.745236552s) [upload=576263]: Next returned nil segment Jan 2 01:17:28.874171721 (657h48m28.745254756s) [upload=576263]: Waiting for error group managing segments... Jan 2 01:17:29.258479519 (657h48m29.129562554s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.269923079 (657h48m29.141006124s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.270658990 (657h48m29.141742035s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.275833464 (657h48m29.146916499s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.333853501 (657h48m29.204936546s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.336841931 (657h48m29.207924976s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.547172416 (657h48m29.418255451s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.564534545 (657h48m29.43561759s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.605074969 (657h48m29.476158014s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.761233684 (657h48m29.63231673s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.903248778 (657h48m29.774331823s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.668470652 (657h48m30.539553687s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868287618 (657h48m30.739370653s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868315942 (657h48m30.739398988s) [seg_pos={0 0}, upload=576263]: Segment reached optimal threshold of 13 pieces. Jan 2 01:17:30.868553525 (657h48m30.73963658s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868565758 (657h48m30.739648813s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868773343 (657h48m30.739856378s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868779566 (657h48m30.739862611s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868784585 (657h48m30.73986762s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868802339 (657h48m30.739885374s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868831775 (657h48m30.73991481s) [seg_pos={0 0}, upload=576263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:17:30.868912969 (657h48m30.739996004s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:17:30.868918210 (657h48m30.740001245s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:17:31.155796691 (657h48m31.026879726s) [upload=576263]: Done uploading segments... Jan 2 01:17:31.155823673 (657h48m31.026906718s) [upload=576263]: Upload finished. err: <nil> Duration: 6.209863094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X...] duration: 6.062075914s trace: https://jaeger.dev.storj.io/trace/5d8db3215b5f0048 FAIL (2026-01-02 01:19:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:17:27.018977618 (657h48m26.890060663s) [upload=576263]: Starting upload Jan 2 01:17:27.019050036 (657h48m26.890133101s) [upload=576263]: Done starting upload Jan 2 01:17:27.019151209 (657h48m26.890234244s) [upload=576263]: Uploading segments... Jan 2 01:17:27.019322414 (657h48m26.89040546s) [upload=576263]: Got next segment. Inline: false Jan 2 01:17:27.019366899 (657h48m26.890449945s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:17:27.019371849 (657h48m26.890454894s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:17:27.262407008 (657h48m27.133490043s) [seg_pos={0 0}, upload=576263]: Begin upload segment... Jan 2 01:17:27.262468044 (657h48m27.133551089s) [seg_pos={0 0}, upload=576263]: Done begin upload segment. Jan 2 01:17:27.262501878 (657h48m27.133584913s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.262578323 (657h48m27.133661368s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263142959 (657h48m27.134225994s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263550565 (657h48m27.1346336s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263937091 (657h48m27.135020126s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264322504 (657h48m27.13540554s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264717267 (657h48m27.135800312s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265237397 (657h48m27.136320442s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265458929 (657h48m27.136541964s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266082807 (657h48m27.137165872s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266599510 (657h48m27.137682545s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.269500053 (657h48m27.140583088s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.270195437 (657h48m27.141278482s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.271042040 (657h48m27.142125085s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.272118508 (657h48m27.143201544s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.273731761 (657h48m27.144814806s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:28.874153517 (657h48m28.745236552s) [upload=576263]: Next returned nil segment Jan 2 01:17:28.874171721 (657h48m28.745254756s) [upload=576263]: Waiting for error group managing segments... Jan 2 01:17:29.258479519 (657h48m29.129562554s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.269923079 (657h48m29.141006124s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.270658990 (657h48m29.141742035s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.275833464 (657h48m29.146916499s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.333853501 (657h48m29.204936546s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.336841931 (657h48m29.207924976s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.547172416 (657h48m29.418255451s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.564534545 (657h48m29.43561759s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.605074969 (657h48m29.476158014s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.761233684 (657h48m29.63231673s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.903248778 (657h48m29.774331823s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.668470652 (657h48m30.539553687s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868287618 (657h48m30.739370653s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868315942 (657h48m30.739398988s) [seg_pos={0 0}, upload=576263]: Segment reached optimal threshold of 13 pieces. Jan 2 01:17:30.868553525 (657h48m30.73963658s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868565758 (657h48m30.739648813s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868773343 (657h48m30.739856378s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868779566 (657h48m30.739862611s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868784585 (657h48m30.73986762s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868802339 (657h48m30.739885374s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868831775 (657h48m30.73991481s) [seg_pos={0 0}, upload=576263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:17:30.868912969 (657h48m30.739996004s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:17:30.868918210 (657h48m30.740001245s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:17:31.155796691 (657h48m31.026879726s) [upload=576263]: Done uploading segments... Jan 2 01:17:31.155823673 (657h48m31.026906718s) [upload=576263]: Upload finished. err: <nil> Duration: 6.062104207s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.233130099s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:19:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.233167219s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 39.088688797s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 39.088716159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 154.888397ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:19:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 154.914697ms core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.436041336s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:19:22 UTC) orbiter warning: context deadline exceeded Duration: 3.436065482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.153160206s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:19:14 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.153212787s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X..] duration: 5.906532138s trace: https://jaeger.dev.storj.io/trace/5d8db3215b5f0048 FAIL (2026-01-02 01:18:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:17:27.018977618 (657h48m26.890060663s) [upload=576263]: Starting upload Jan 2 01:17:27.019050036 (657h48m26.890133101s) [upload=576263]: Done starting upload Jan 2 01:17:27.019151209 (657h48m26.890234244s) [upload=576263]: Uploading segments... Jan 2 01:17:27.019322414 (657h48m26.89040546s) [upload=576263]: Got next segment. Inline: false Jan 2 01:17:27.019366899 (657h48m26.890449945s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:17:27.019371849 (657h48m26.890454894s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:17:27.262407008 (657h48m27.133490043s) [seg_pos={0 0}, upload=576263]: Begin upload segment... Jan 2 01:17:27.262468044 (657h48m27.133551089s) [seg_pos={0 0}, upload=576263]: Done begin upload segment. Jan 2 01:17:27.262501878 (657h48m27.133584913s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.262578323 (657h48m27.133661368s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263142959 (657h48m27.134225994s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263550565 (657h48m27.1346336s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263937091 (657h48m27.135020126s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264322504 (657h48m27.13540554s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264717267 (657h48m27.135800312s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265237397 (657h48m27.136320442s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265458929 (657h48m27.136541964s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266082807 (657h48m27.137165872s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266599510 (657h48m27.137682545s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.269500053 (657h48m27.140583088s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.270195437 (657h48m27.141278482s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.271042040 (657h48m27.142125085s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.272118508 (657h48m27.143201544s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.273731761 (657h48m27.144814806s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:28.874153517 (657h48m28.745236552s) [upload=576263]: Next returned nil segment Jan 2 01:17:28.874171721 (657h48m28.745254756s) [upload=576263]: Waiting for error group managing segments... Jan 2 01:17:29.258479519 (657h48m29.129562554s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.269923079 (657h48m29.141006124s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.270658990 (657h48m29.141742035s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.275833464 (657h48m29.146916499s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.333853501 (657h48m29.204936546s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.336841931 (657h48m29.207924976s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.547172416 (657h48m29.418255451s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.564534545 (657h48m29.43561759s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.605074969 (657h48m29.476158014s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.761233684 (657h48m29.63231673s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.903248778 (657h48m29.774331823s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.668470652 (657h48m30.539553687s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868287618 (657h48m30.739370653s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868315942 (657h48m30.739398988s) [seg_pos={0 0}, upload=576263]: Segment reached optimal threshold of 13 pieces. Jan 2 01:17:30.868553525 (657h48m30.73963658s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868565758 (657h48m30.739648813s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868773343 (657h48m30.739856378s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868779566 (657h48m30.739862611s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868784585 (657h48m30.73986762s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868802339 (657h48m30.739885374s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868831775 (657h48m30.73991481s) [seg_pos={0 0}, upload=576263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:17:30.868912969 (657h48m30.739996004s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:17:30.868918210 (657h48m30.740001245s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:17:31.155796691 (657h48m31.026879726s) [upload=576263]: Done uploading segments... Jan 2 01:17:31.155823673 (657h48m31.026906718s) [upload=576263]: Upload finished. err: <nil> Duration: 5.906556293s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 151.287399ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:18:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 151.333677ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 29.989056395s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:18:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 29.98907401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.592864397s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:18:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.592894934s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X.] duration: 11.100389377s trace: https://jaeger.dev.storj.io/trace/5d8db3215b5f0048 FAIL (2026-01-02 01:18:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:17:27.018977618 (657h48m26.890060663s) [upload=576263]: Starting upload Jan 2 01:17:27.019050036 (657h48m26.890133101s) [upload=576263]: Done starting upload Jan 2 01:17:27.019151209 (657h48m26.890234244s) [upload=576263]: Uploading segments... Jan 2 01:17:27.019322414 (657h48m26.89040546s) [upload=576263]: Got next segment. Inline: false Jan 2 01:17:27.019366899 (657h48m26.890449945s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:17:27.019371849 (657h48m26.890454894s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:17:27.262407008 (657h48m27.133490043s) [seg_pos={0 0}, upload=576263]: Begin upload segment... Jan 2 01:17:27.262468044 (657h48m27.133551089s) [seg_pos={0 0}, upload=576263]: Done begin upload segment. Jan 2 01:17:27.262501878 (657h48m27.133584913s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.262578323 (657h48m27.133661368s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263142959 (657h48m27.134225994s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263550565 (657h48m27.1346336s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263937091 (657h48m27.135020126s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264322504 (657h48m27.13540554s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264717267 (657h48m27.135800312s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265237397 (657h48m27.136320442s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265458929 (657h48m27.136541964s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266082807 (657h48m27.137165872s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266599510 (657h48m27.137682545s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.269500053 (657h48m27.140583088s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.270195437 (657h48m27.141278482s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.271042040 (657h48m27.142125085s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.272118508 (657h48m27.143201544s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.273731761 (657h48m27.144814806s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:28.874153517 (657h48m28.745236552s) [upload=576263]: Next returned nil segment Jan 2 01:17:28.874171721 (657h48m28.745254756s) [upload=576263]: Waiting for error group managing segments... Jan 2 01:17:29.258479519 (657h48m29.129562554s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.269923079 (657h48m29.141006124s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.270658990 (657h48m29.141742035s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.275833464 (657h48m29.146916499s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.333853501 (657h48m29.204936546s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.336841931 (657h48m29.207924976s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.547172416 (657h48m29.418255451s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.564534545 (657h48m29.43561759s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.605074969 (657h48m29.476158014s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.761233684 (657h48m29.63231673s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.903248778 (657h48m29.774331823s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.668470652 (657h48m30.539553687s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868287618 (657h48m30.739370653s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868315942 (657h48m30.739398988s) [seg_pos={0 0}, upload=576263]: Segment reached optimal threshold of 13 pieces. Jan 2 01:17:30.868553525 (657h48m30.73963658s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868565758 (657h48m30.739648813s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868773343 (657h48m30.739856378s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868779566 (657h48m30.739862611s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868784585 (657h48m30.73986762s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868802339 (657h48m30.739885374s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868831775 (657h48m30.73991481s) [seg_pos={0 0}, upload=576263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:17:30.868912969 (657h48m30.739996004s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:17:30.868918210 (657h48m30.740001245s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:17:31.155796691 (657h48m31.026879726s) [upload=576263]: Done uploading segments... Jan 2 01:17:31.155823673 (657h48m31.026906718s) [upload=576263]: Upload finished. err: <nil> Duration: 11.100410559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.697325591s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:18:35 UTC) orbiter warning: context deadline exceeded Duration: 3.697348104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 25.013169632s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 25.013202805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.54827318s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:17:57 UTC) orbiter warning: context deadline exceeded Duration: 3.548303417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....X] duration: 30.002507921s trace: https://jaeger.dev.storj.io/trace/5d8db3215b5f0048 FAIL (2026-01-02 01:17:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:17:27.018977618 (657h48m26.890060663s) [upload=576263]: Starting upload Jan 2 01:17:27.019050036 (657h48m26.890133101s) [upload=576263]: Done starting upload Jan 2 01:17:27.019151209 (657h48m26.890234244s) [upload=576263]: Uploading segments... Jan 2 01:17:27.019322414 (657h48m26.89040546s) [upload=576263]: Got next segment. Inline: false Jan 2 01:17:27.019366899 (657h48m26.890449945s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:17:27.019371849 (657h48m26.890454894s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:17:27.262407008 (657h48m27.133490043s) [seg_pos={0 0}, upload=576263]: Begin upload segment... Jan 2 01:17:27.262468044 (657h48m27.133551089s) [seg_pos={0 0}, upload=576263]: Done begin upload segment. Jan 2 01:17:27.262501878 (657h48m27.133584913s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.262578323 (657h48m27.133661368s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263142959 (657h48m27.134225994s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263550565 (657h48m27.1346336s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.263937091 (657h48m27.135020126s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264322504 (657h48m27.13540554s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.264717267 (657h48m27.135800312s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265237397 (657h48m27.136320442s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.265458929 (657h48m27.136541964s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266082807 (657h48m27.137165872s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.266599510 (657h48m27.137682545s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.269500053 (657h48m27.140583088s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.270195437 (657h48m27.141278482s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.271042040 (657h48m27.142125085s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.272118508 (657h48m27.143201544s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:27.273731761 (657h48m27.144814806s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Uploading piece... Jan 2 01:17:28.874153517 (657h48m28.745236552s) [upload=576263]: Next returned nil segment Jan 2 01:17:28.874171721 (657h48m28.745254756s) [upload=576263]: Waiting for error group managing segments... Jan 2 01:17:29.258479519 (657h48m29.129562554s) [noise=true , address=145.79.227.6:4259 , piece_id=5DDFVLGWUMGBZQ566QBGLK2EWVQYHXMN6GONENSECFBTG3J5ZKCA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.269923079 (657h48m29.141006124s) [noise=true , address=145.79.227.41:4085 , piece_id=3S7TUKFX6LDGYN33WOPJJ5HFDZG6OARK2IYEY2PTMVVSSRUFOFEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.270658990 (657h48m29.141742035s) [noise=true , address=145.79.227.48:4301 , piece_id=YBFGJQJIO7G4RFERUXBIPZTNA66VXYOEJPWZ6W23JR66N7YQPCEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.275833464 (657h48m29.146916499s) [noise=true , address=145.79.227.12:4765 , piece_id=YITDC3462GBMMIHDWDBACJN6S7CUR36GRZXYLC6MEKPY2LB3X3CQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.333853501 (657h48m29.204936546s) [noise=true , address=145.79.227.52:4445 , piece_id=RTV3VGD2ZYSDBM76IQO3LBIP6PPTSBKK6UHT3BFNUPPEMOGBBBNA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.336841931 (657h48m29.207924976s) [noise=true , address=145.79.227.2:4114 , piece_id=WRWUW6SQ6NAPABK7FKXOHYDVUQ3WDEBVPRPJWQUZVXTXMBSFA3DA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.547172416 (657h48m29.418255451s) [noise=true , address=77.90.31.15:4697 , piece_id=LCP5VWT7ADHTE7DHUKLJBZ5ITOVM4X6KRSRVKKN5RJAZPQHJCYRQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.564534545 (657h48m29.43561759s) [noise=true , address=45.91.116.6:4973 , piece_id=A5KZVKZP2IY5H75G52U6P6WIVTX2VGWC46B6SL7V7GHQWGJVJ73Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.605074969 (657h48m29.476158014s) [noise=true , address=45.91.116.5:4297 , piece_id=U72YJ5QLH5PB5MUCBLLNBJY3BGV63LQKYKCKDNAPD6IZAS2LZAFQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.761233684 (657h48m29.63231673s) [noise=true , address=45.91.116.7:4010 , piece_id=BLXEMGDU57M43W4MZ3XCSR3BLPFNQUGKCOK3ASOSPOKWPNRKYF4Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:29.903248778 (657h48m29.774331823s) [noise=true , address=45.91.116.8:4048 , piece_id=FPWOKRWBDDYBHJHLTJT2N3H5AIXDVTPQPLPML7FN4DMXBOMET3MQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.668470652 (657h48m30.539553687s) [noise=true , address=45.91.116.4:4940 , piece_id=RJNMAKFQOKAHQK3NI42DAKFM66LUEH5YPMYWSIEHVF3XT6JNMQGA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868287618 (657h48m30.739370653s) [noise=true , address=145.79.227.23:4545 , piece_id=LK7JZT3NKIHIVB6NMYFPDJZTIYCC4DCKWM2X6AOMBOU6AKGGQPEQ, seg_pos={0 0}, upload=576263]: Done uploading piece. err: <nil> Jan 2 01:17:30.868315942 (657h48m30.739398988s) [seg_pos={0 0}, upload=576263]: Segment reached optimal threshold of 13 pieces. Jan 2 01:17:30.868553525 (657h48m30.73963658s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868565758 (657h48m30.739648813s) [noise=true , address=45.91.116.11:4119 , piece_id=YCQPJJIAZIHUKPLTOCIRUX7U3253ZHA5XY24NOG3ON3KVVFWSR6Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868773343 (657h48m30.739856378s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868779566 (657h48m30.739862611s) [noise=true , address=145.79.227.49:4338 , piece_id=MAKJLS3XHVTP2FNCFUYXKG64JZJDEWMOFINZF5GADZO5V5Z3Y43Q, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868784585 (657h48m30.73986762s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Jan 2 01:17:30.868802339 (657h48m30.739885374s) [noise=true , address=145.79.227.20:4437 , piece_id=37K2YDVJKBDQJQLPC27UBHLCCXV54OAZQDHTVFFZQFKRELJETJJA, seg_pos={0 0}, upload=576263]: Long tail cancellation due to optimal threshold Jan 2 01:17:30.868831775 (657h48m30.73991481s) [seg_pos={0 0}, upload=576263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:17:30.868912969 (657h48m30.739996004s) [upload=576263]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:17:30.868918210 (657h48m30.740001245s) [upload=576263]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:17:31.155796691 (657h48m31.026879726s) [upload=576263]: Done uploading segments... Jan 2 01:17:31.155823673 (657h48m31.026906718s) [upload=576263]: Upload finished. err: <nil> Duration: 30.002520986s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 153.224227ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:17:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 153.25753ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.574988558s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:17:45 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.57503193s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 35.317639729s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:17:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 35.317754228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.478451481s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:17:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.478480076s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.535052626s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:17:20 UTC) orbiter warning: context deadline exceeded Duration: 3.53507514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.580623157s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:17:01 UTC) orbiter warning: context deadline exceeded Duration: 3.580662853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 190.957767ms trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:16:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 190.992333ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX....] duration: 5.905599545s trace: https://jaeger.dev.storj.io/trace/5ad88850092560c3 FAIL (2026-01-02 01:16:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:13:57.679063066 (657h44m57.550146111s) [upload=576214]: Starting upload Jan 2 01:13:57.679161463 (657h44m57.550244508s) [upload=576214]: Done starting upload Jan 2 01:13:57.679501981 (657h44m57.550585026s) [upload=576214]: Uploading segments... Jan 2 01:13:57.680723607 (657h44m57.551806653s) [upload=576214]: Got next segment. Inline: false Jan 2 01:13:57.680796055 (657h44m57.5518791s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:13:57.680802918 (657h44m57.551885963s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:13:57.927753733 (657h44m57.798836778s) [seg_pos={0 0}, upload=576214]: Begin upload segment... Jan 2 01:13:57.927824538 (657h44m57.798907573s) [seg_pos={0 0}, upload=576214]: Done begin upload segment. Jan 2 01:13:57.927858112 (657h44m57.798941157s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928188210 (657h44m57.799271255s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928815494 (657h44m57.799898539s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928868986 (657h44m57.799952031s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929256304 (657h44m57.800339339s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929397192 (657h44m57.800480237s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929622250 (657h44m57.800705296s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930077678 (657h44m57.801160723s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930115630 (657h44m57.801198685s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930171927 (657h44m57.801254972s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930514850 (657h44m57.801597896s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930783612 (657h44m57.801866657s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931336805 (657h44m57.80241985s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931905048 (657h44m57.802988083s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.933228888 (657h44m57.804311934s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.934221248 (657h44m57.805304283s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:59.628878079 (657h44m59.499961114s) [upload=576214]: Next returned nil segment Jan 2 01:13:59.628891153 (657h44m59.499974199s) [upload=576214]: Waiting for error group managing segments... Jan 2 01:13:59.916483174 (657h44m59.787566219s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.921936066 (657h44m59.793019111s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.922565705 (657h44m59.79364874s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.949948206 (657h44m59.821031251s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.956974033 (657h44m59.828057068s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.984721057 (657h44m59.855804092s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999124042 (657h44m59.870207077s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999691733 (657h44m59.870774768s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001642107 (657h44m59.872725162s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001893215 (657h44m59.87297625s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.102657523 (657h44m59.973740568s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.221905184 (657h45m0.092988229s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279415310 (657h45m0.150498345s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279436261 (657h45m0.150519296s) [seg_pos={0 0}, upload=576214]: Segment reached optimal threshold of 13 pieces. Jan 2 01:14:00.279589773 (657h45m0.150672808s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279596165 (657h45m0.1506792s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279686767 (657h45m0.150769813s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279694913 (657h45m0.150777948s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279813839 (657h45m0.150896884s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279820482 (657h45m0.150903527s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279836081 (657h45m0.150919117s) [seg_pos={0 0}, upload=576214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:14:00.279903780 (657h45m0.150986826s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:14:00.279908339 (657h45m0.150991384s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:14:00.574385118 (657h45m0.445468163s) [upload=576214]: Done uploading segments... Jan 2 01:14:00.574404145 (657h45m0.44548718s) [upload=576214]: Upload finished. err: <nil> Duration: 5.905629303s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000840577s trace: https://jaeger.dev.storj.io/trace/8065ca37104e4f4 FAIL (2026-01-02 01:16:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000866104s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 24.253653705s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:16:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 24.253678874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.145331982s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:16:12 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.14538358s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.578961753s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:16:10 UTC) orbiter warning: context deadline exceeded Duration: 3.578989606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX...] duration: 5.745105823s trace: https://jaeger.dev.storj.io/trace/5ad88850092560c3 FAIL (2026-01-02 01:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:13:57.679063066 (657h44m57.550146111s) [upload=576214]: Starting upload Jan 2 01:13:57.679161463 (657h44m57.550244508s) [upload=576214]: Done starting upload Jan 2 01:13:57.679501981 (657h44m57.550585026s) [upload=576214]: Uploading segments... Jan 2 01:13:57.680723607 (657h44m57.551806653s) [upload=576214]: Got next segment. Inline: false Jan 2 01:13:57.680796055 (657h44m57.5518791s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:13:57.680802918 (657h44m57.551885963s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:13:57.927753733 (657h44m57.798836778s) [seg_pos={0 0}, upload=576214]: Begin upload segment... Jan 2 01:13:57.927824538 (657h44m57.798907573s) [seg_pos={0 0}, upload=576214]: Done begin upload segment. Jan 2 01:13:57.927858112 (657h44m57.798941157s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928188210 (657h44m57.799271255s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928815494 (657h44m57.799898539s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928868986 (657h44m57.799952031s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929256304 (657h44m57.800339339s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929397192 (657h44m57.800480237s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929622250 (657h44m57.800705296s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930077678 (657h44m57.801160723s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930115630 (657h44m57.801198685s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930171927 (657h44m57.801254972s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930514850 (657h44m57.801597896s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930783612 (657h44m57.801866657s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931336805 (657h44m57.80241985s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931905048 (657h44m57.802988083s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.933228888 (657h44m57.804311934s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.934221248 (657h44m57.805304283s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:59.628878079 (657h44m59.499961114s) [upload=576214]: Next returned nil segment Jan 2 01:13:59.628891153 (657h44m59.499974199s) [upload=576214]: Waiting for error group managing segments... Jan 2 01:13:59.916483174 (657h44m59.787566219s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.921936066 (657h44m59.793019111s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.922565705 (657h44m59.79364874s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.949948206 (657h44m59.821031251s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.956974033 (657h44m59.828057068s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.984721057 (657h44m59.855804092s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999124042 (657h44m59.870207077s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999691733 (657h44m59.870774768s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001642107 (657h44m59.872725162s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001893215 (657h44m59.87297625s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.102657523 (657h44m59.973740568s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.221905184 (657h45m0.092988229s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279415310 (657h45m0.150498345s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279436261 (657h45m0.150519296s) [seg_pos={0 0}, upload=576214]: Segment reached optimal threshold of 13 pieces. Jan 2 01:14:00.279589773 (657h45m0.150672808s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279596165 (657h45m0.1506792s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279686767 (657h45m0.150769813s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279694913 (657h45m0.150777948s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279813839 (657h45m0.150896884s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279820482 (657h45m0.150903527s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279836081 (657h45m0.150919117s) [seg_pos={0 0}, upload=576214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:14:00.279903780 (657h45m0.150986826s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:14:00.279908339 (657h45m0.150991384s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:14:00.574385118 (657h45m0.445468163s) [upload=576214]: Done uploading segments... Jan 2 01:14:00.574404145 (657h45m0.44548718s) [upload=576214]: Upload finished. err: <nil> Duration: 5.745136861s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.371975408s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:15:40 UTC) orbiter warning: context deadline exceeded Duration: 3.37200258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 38.835080385s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:15:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 38.8350987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX..] duration: 5.967872762s trace: https://jaeger.dev.storj.io/trace/5ad88850092560c3 FAIL (2026-01-02 01:15:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:13:57.679063066 (657h44m57.550146111s) [upload=576214]: Starting upload Jan 2 01:13:57.679161463 (657h44m57.550244508s) [upload=576214]: Done starting upload Jan 2 01:13:57.679501981 (657h44m57.550585026s) [upload=576214]: Uploading segments... Jan 2 01:13:57.680723607 (657h44m57.551806653s) [upload=576214]: Got next segment. Inline: false Jan 2 01:13:57.680796055 (657h44m57.5518791s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:13:57.680802918 (657h44m57.551885963s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:13:57.927753733 (657h44m57.798836778s) [seg_pos={0 0}, upload=576214]: Begin upload segment... Jan 2 01:13:57.927824538 (657h44m57.798907573s) [seg_pos={0 0}, upload=576214]: Done begin upload segment. Jan 2 01:13:57.927858112 (657h44m57.798941157s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928188210 (657h44m57.799271255s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928815494 (657h44m57.799898539s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928868986 (657h44m57.799952031s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929256304 (657h44m57.800339339s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929397192 (657h44m57.800480237s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929622250 (657h44m57.800705296s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930077678 (657h44m57.801160723s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930115630 (657h44m57.801198685s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930171927 (657h44m57.801254972s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930514850 (657h44m57.801597896s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930783612 (657h44m57.801866657s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931336805 (657h44m57.80241985s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931905048 (657h44m57.802988083s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.933228888 (657h44m57.804311934s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.934221248 (657h44m57.805304283s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:59.628878079 (657h44m59.499961114s) [upload=576214]: Next returned nil segment Jan 2 01:13:59.628891153 (657h44m59.499974199s) [upload=576214]: Waiting for error group managing segments... Jan 2 01:13:59.916483174 (657h44m59.787566219s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.921936066 (657h44m59.793019111s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.922565705 (657h44m59.79364874s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.949948206 (657h44m59.821031251s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.956974033 (657h44m59.828057068s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.984721057 (657h44m59.855804092s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999124042 (657h44m59.870207077s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999691733 (657h44m59.870774768s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001642107 (657h44m59.872725162s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001893215 (657h44m59.87297625s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.102657523 (657h44m59.973740568s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.221905184 (657h45m0.092988229s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279415310 (657h45m0.150498345s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279436261 (657h45m0.150519296s) [seg_pos={0 0}, upload=576214]: Segment reached optimal threshold of 13 pieces. Jan 2 01:14:00.279589773 (657h45m0.150672808s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279596165 (657h45m0.1506792s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279686767 (657h45m0.150769813s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279694913 (657h45m0.150777948s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279813839 (657h45m0.150896884s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279820482 (657h45m0.150903527s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279836081 (657h45m0.150919117s) [seg_pos={0 0}, upload=576214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:14:00.279903780 (657h45m0.150986826s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:14:00.279908339 (657h45m0.150991384s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:14:00.574385118 (657h45m0.445468163s) [upload=576214]: Done uploading segments... Jan 2 01:14:00.574404145 (657h45m0.44548718s) [upload=576214]: Upload finished. err: <nil> Duration: 5.967900134s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 3.197789988s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:15:04 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.197824283s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.560245218s trace: https://jaeger.dev.storj.io/trace/7df28793b9d9c02f FAIL (2026-01-02 01:14:55 UTC) orbiter warning: context deadline exceeded Duration: 3.56027226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005286324s trace: https://jaeger.dev.storj.io/trace/5cfe4ac2a06b7d06 FAIL (2026-01-02 01:14:51 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005309357s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 33.053366356s trace: https://jaeger.dev.storj.io/trace/3227d9fcecdd3773 FAIL (2026-01-02 01:14:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697165214 (657h7m37.568248249s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.697587017 (657h7m37.568670052s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698049198 (657h7m37.569132233s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.698607942 (657h7m37.569690977s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699200520 (657h7m37.570283565s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.699585624 (657h7m37.570668659s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.700115232 (657h7m37.571198297s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.701631649 (657h7m37.572714684s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.703275769 (657h7m37.574358804s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.704428584 (657h7m37.575511619s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.705545030 (657h7m37.576628075s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.708602091 (657h7m37.579685126s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.711724127 (657h7m37.582807172s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:37.715235752 (657h7m37.586318798s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Uploading piece... Jan 2 00:36:39.114765832 (657h7m38.985848867s) [upload=575662]: Next returned nil segment Jan 2 00:36:39.114777704 (657h7m38.985860739s) [upload=575662]: Waiting for error group managing segments... Jan 2 00:36:39.587317829 (657h7m39.458400874s) [noise=true , address=64.49.13.137:4499 , piece_id=OVTJTP2VFU6BGU2WQZ5IBEPKI2XYJ5SGM75OZUI4FKHKMA54OUVA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.592964281 (657h7m39.464047326s) [noise=true , address=145.79.227.37:4572 , piece_id=I75FQUDP66QKRKB3MPVY3EATUGSTVMB7OCIMBIEEKZOPHVJUTOEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.604274326 (657h7m39.475357361s) [noise=true , address=145.79.227.11:4763 , piece_id=4HBOFZRPKTB5RB4WTMMZVMXABVTMP4QQIB63B23ZF3MOZ7CKHOLA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.798444911 (657h7m39.669527946s) [noise=true , address=77.90.31.10:4532 , piece_id=LCNGLXCA6HRCCLYVDL2EAAZINOIS5XMXKNV26Q3LH2HSLVOXEFEQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.824981409 (657h7m39.696064445s) [noise=true , address=77.90.31.16:4744 , piece_id=KCRJG73DAXBI5UENQATHNYNSP3BIQ25H24WFBKNPY4UECH55DOJQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.877452610 (657h7m39.748535645s) [noise=true , address=77.90.31.12:4606 , piece_id=LTLFFF6GUASRHCGTSWVZZ6OX44GTQZGCWS5YXFXZ6TGD5GJO6GWA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.882510131 (657h7m39.753593177s) [noise=true , address=45.91.116.7:4040 , piece_id=FTGUCJY3OUBQBTJZBQZW4QHK5K74RPAFCZABNKN44UOGDA5QJEDQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:39.885310293 (657h7m39.756393328s) [noise=true , address=77.90.31.21:4901 , piece_id=WDQT33ANAMWAGGNFM27BQ6BT5UXIDX63GED4L4IJICYZB73JSRRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.007567244 (657h7m39.878650279s) [noise=true , address=77.90.31.15:4732 , piece_id=CYHQ5REQ7GFBVK5QXNV7T72XQQER65OFVFRQQP44BJQHC6BGSANQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.171420692 (657h7m40.042503727s) [noise=true , address=77.90.31.19:4859 , piece_id=SEUW6PBBTUIXDZYEH7A54I6RBYAUA344HRHS37LEFTILTTDCMRPQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.376562367 (657h7m40.247645402s) [noise=true , address=193.149.164.143:4696 , piece_id=K75KF5TD3LUAMY5ODI7H3LTHOD3GSAWUDBEU47B4PMMZFIQTL24A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.386296402 (657h7m40.257379437s) [noise=true , address=45.91.116.8:4072 , piece_id=6RV6QZLWONEXDQUSADRGB2LKWRCJIO3KA23CZKF7S5K6BEWAUIYQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.586543066 (657h7m40.457626111s) [noise=true , address=145.79.227.44:4226 , piece_id=ITAD4OHN3FWKZN4JADTEWU3KHK6IHNDDKK5RY36ZFQWTBJNMHBFQ, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627808881 (657h7m40.498891916s) [noise=true , address=145.79.227.34:4913 , piece_id=6OVR6I3CWHGUWH6UHXIVEYLZMV3I4UPZ5IPODGZMCJGKRBBIFZRA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: <nil> Jan 2 00:36:40.627832386 (657h7m40.498915421s) [seg_pos={0 0}, upload=575662]: Segment reached optimal threshold of 14 pieces. Jan 2 00:36:40.628014282 (657h7m40.499097327s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1obTuzA1eVY69zki5vcH4cUjgZJzXnbKaj17TLjCLhDhLi7ZEi): protocol: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51232->145.79.227.14:4850: use of closed network connection Jan 2 00:36:40.628020414 (657h7m40.499103449s) [noise=true , address=145.79.227.14:4850 , piece_id=XYLE3RXJARNV3WZWM6FP6S4MQWFPOYYOVSIGUA6U3RH4TFSL7STA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628073034 (657h7m40.49915612s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53026->145.79.227.35:4487: use of closed network connection Jan 2 00:36:40.628083825 (657h7m40.49916688s) [noise=true , address=145.79.227.35:4487 , piece_id=PPSJDNR4OEV6MIALV4L3Z3DY46BSYDTDLYUC25QJM35KJ5NMNX2A, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628284727 (657h7m40.499367762s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5KYLZKVjdYfsKyPN4nN82usmg1frD8d9SsijTmRHVxPdWe68): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628293945 (657h7m40.49937699s) [noise=true , address=145.79.227.23:4579 , piece_id=GKDASFPIZ6DJQHPQ7ORQ5FHRI4IFLGWSB5UKC5DTNSDIWXQ66UMA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628318682 (657h7m40.499401727s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim): protocol: expected piece hash; context canceled; EOF Jan 2 00:36:40.628329623 (657h7m40.499412668s) [noise=true , address=145.79.227.34:4938 , piece_id=UY7AMZBVHQII25L6RUPYQF6C6C2G7O4NXG75PYGDYFJTV2VVPNBA, seg_pos={0 0}, upload=575662]: Long tail cancellation due to optimal threshold Jan 2 00:36:40.628352557 (657h7m40.499435592s) [seg_pos={0 0}, upload=575662]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 2 00:36:40.628434953 (657h7m40.499517988s) [upload=575662]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 00:36:40.628440243 (657h7m40.499523288s) [upload=575662]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 00:36:40.936781491 (657h7m40.807864537s) [upload=575662]: Done uploading segments... Jan 2 00:36:40.936806069 (657h7m40.807889114s) [upload=575662]: Upload finished. err: <nil> Duration: 33.053396322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....X.XX......X...X....XX.X..X....XXX....X.XX...X..X..XXXX.] duration: 6.788101493s trace: https://jaeger.dev.storj.io/trace/5ad88850092560c3 FAIL (2026-01-02 01:14:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 2 01:13:57.679063066 (657h44m57.550146111s) [upload=576214]: Starting upload Jan 2 01:13:57.679161463 (657h44m57.550244508s) [upload=576214]: Done starting upload Jan 2 01:13:57.679501981 (657h44m57.550585026s) [upload=576214]: Uploading segments... Jan 2 01:13:57.680723607 (657h44m57.551806653s) [upload=576214]: Got next segment. Inline: false Jan 2 01:13:57.680796055 (657h44m57.5518791s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 2 01:13:57.680802918 (657h44m57.551885963s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 2 01:13:57.927753733 (657h44m57.798836778s) [seg_pos={0 0}, upload=576214]: Begin upload segment... Jan 2 01:13:57.927824538 (657h44m57.798907573s) [seg_pos={0 0}, upload=576214]: Done begin upload segment. Jan 2 01:13:57.927858112 (657h44m57.798941157s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928188210 (657h44m57.799271255s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928815494 (657h44m57.799898539s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.928868986 (657h44m57.799952031s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929256304 (657h44m57.800339339s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929397192 (657h44m57.800480237s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.929622250 (657h44m57.800705296s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930077678 (657h44m57.801160723s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930115630 (657h44m57.801198685s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930171927 (657h44m57.801254972s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930514850 (657h44m57.801597896s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.930783612 (657h44m57.801866657s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931336805 (657h44m57.80241985s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.931905048 (657h44m57.802988083s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.933228888 (657h44m57.804311934s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:57.934221248 (657h44m57.805304283s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Uploading piece... Jan 2 01:13:59.628878079 (657h44m59.499961114s) [upload=576214]: Next returned nil segment Jan 2 01:13:59.628891153 (657h44m59.499974199s) [upload=576214]: Waiting for error group managing segments... Jan 2 01:13:59.916483174 (657h44m59.787566219s) [noise=true , address=145.79.227.7:4294 , piece_id=IHWNBJFFMWHIKEFQSPOMH5DY4EFNFFCYBWPSHJ4CZDRX6SV363XQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.921936066 (657h44m59.793019111s) [noise=true , address=145.79.227.39:4013 , piece_id=AKSCLFBL6FPBM4BZLTUUIK6PZH7R2PG3NUIJIMY2763Q73GK2AQQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.922565705 (657h44m59.79364874s) [noise=true , address=145.79.227.10:4401 , piece_id=IAGYS6YHYAHA7MYJF2E3KUOVK7BQIFAABZPK6AK32V5KLTH7JJTA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.949948206 (657h44m59.821031251s) [noise=true , address=145.79.227.42:4121 , piece_id=TVEXPPHVOV6N3W52ZNRSR6ZO5BCNWTF4E4MBJCSDCYNWRYBHVXJQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.956974033 (657h44m59.828057068s) [noise=true , address=145.79.227.21:4474 , piece_id=PBNECXZFDKFHP2UGQDEOOHVGS4KWEYCUCW75OTFRLVJPKY5X62KA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.984721057 (657h44m59.855804092s) [noise=true , address=145.79.227.28:4725 , piece_id=RCABD4I5IIKJGW2MDO2LT7KWVF5R7G6FTUYYXYJTOTCQEMXIGKWQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999124042 (657h44m59.870207077s) [noise=true , address=145.79.227.48:4302 , piece_id=LEKQLMH5QYUJX6Q5WCFBS7GUN7YRT6PPMMUSYHW3QLH6C2KMRPLA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:13:59.999691733 (657h44m59.870774768s) [noise=true , address=145.79.227.31:4797 , piece_id=M5JAB6J3R2NUOFW3IHAE2XTBCMXBKGWB6TVG3F3WEVARDWON6MPQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001642107 (657h44m59.872725162s) [noise=true , address=145.79.227.13:4802 , piece_id=4LWYEL2SDJ3ECG5SHXNFQIQ5QRQSNG5LNH2I4L4W2G2VOUXPLCPA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.001893215 (657h44m59.87297625s) [noise=true , address=145.79.227.36:4518 , piece_id=67OCJZV4OQHY3AOYRA7P2Z7MDZJXL22MVFNM2ZWHK2WTD35YWAOA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.102657523 (657h44m59.973740568s) [noise=true , address=77.90.31.21:4878 , piece_id=SXQN7BFXCA2VI3CMT4THCUB2ZR6WAIK4ORKWWEPYHEDHSDKGT5JA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.221905184 (657h45m0.092988229s) [noise=true , address=77.90.31.17:4770 , piece_id=LLXOZ4I7TCOMODCNCQONJXBZDHB3SZXEX3XQSHIKQVTJ2VH5CN5A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279415310 (657h45m0.150498345s) [noise=true , address=45.91.116.13:4156 , piece_id=JZQNINESSREX5MDKTJEPG7NOCX4MR7QSHDI6GUMJME3MRQP6AIOQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: <nil> Jan 2 01:14:00.279436261 (657h45m0.150519296s) [seg_pos={0 0}, upload=576214]: Segment reached optimal threshold of 13 pieces. Jan 2 01:14:00.279589773 (657h45m0.150672808s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279596165 (657h45m0.1506792s) [noise=true , address=145.79.227.33:4871 , piece_id=CZMB3KCT7O3HJIZ3Z3OBSGZQT4JAMPEHQJBY3ZDUDDIPHJJTWUAA, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279686767 (657h45m0.150769813s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279694913 (657h45m0.150777948s) [noise=true , address=77.90.31.10:4519 , piece_id=7YXC7B76Y433CFXD6KPABEGI2QV5GR2MA66RZ2BE4Y6GMKPGAH2A, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279813839 (657h45m0.150896884s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 2 01:14:00.279820482 (657h45m0.150903527s) [noise=true , address=45.91.116.9:4264 , piece_id=TTITTLLUV7QMEABJJH4G2YCVRMODHNSFAY4G62UZFEEXLNT3WFBQ, seg_pos={0 0}, upload=576214]: Long tail cancellation due to optimal threshold Jan 2 01:14:00.279836081 (657h45m0.150919117s) [seg_pos={0 0}, upload=576214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 2 01:14:00.279903780 (657h45m0.150986826s) [upload=576214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 2 01:14:00.279908339 (657h45m0.150991384s) [upload=576214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 2 01:14:00.574385118 (657h45m0.445468163s) [upload=576214]: Done uploading segments... Jan 2 01:14:00.574404145 (657h45m0.44548718s) [upload=576214]: Upload finished. err: <nil> Duration: 6.788127844s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)