Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X......X.....X.X....X.X......X......X.XX..X....X..X..] duration: 5.904392947s trace: https://jaeger.dev.storj.io/trace/1a2df166b5f1cebe FAIL (2025-12-27 08:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A) context deadline exceeded ======LOGS====== Dec 27 08:18:03.171091985 (520h49m3.04217503s) [upload=456900]: Starting upload Dec 27 08:18:03.171179261 (520h49m3.042262307s) [upload=456900]: Done starting upload Dec 27 08:18:03.171312204 (520h49m3.042395239s) [upload=456900]: Uploading segments... Dec 27 08:18:03.171468322 (520h49m3.042551357s) [upload=456900]: Got next segment. Inline: false Dec 27 08:18:03.171506925 (520h49m3.04258997s) [upload=456900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:18:03.171514399 (520h49m3.042597434s) [upload=456900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:18:03.414596757 (520h49m3.285679812s) [seg_pos={0 0}, upload=456900]: Begin upload segment... Dec 27 08:18:03.414669295 (520h49m3.28575234s) [seg_pos={0 0}, upload=456900]: Done begin upload segment. Dec 27 08:18:03.414681088 (520h49m3.285764133s) [noise=true , address=145.79.227.17:4971 , piece_id=C5JBC4TQVGRB743HMZMSNKTWLMDFR5GDTGMCKVUUZSSBH2CEGKMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.414711426 (520h49m3.285794471s) [noise=true , address=145.79.227.27:4689 , piece_id=KJGZYISMDIJ7MQKJVGTVLXPSY765B4J4BSUAIBLO3YXI55GGT6PA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.415309094 (520h49m3.286392139s) [noise=true , address=145.79.227.21:4474 , piece_id=4MLFNK3F5XUY6ZTHEZS2DZFLQGMHE2LQCDBHVPMX2XRHVOT7JYMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.415698856 (520h49m3.286781901s) [noise=true , address=77.90.31.11:4554 , piece_id=EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416086915 (520h49m3.28716996s) [noise=true , address=145.79.227.50:4375 , piece_id=JIWLDRLSKO6U7NZWOG4HYXDM4QRBDLWOH3KR6BGKMQT23RTUERDQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416494040 (520h49m3.287577085s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416913338 (520h49m3.287996384s) [noise=true , address=145.79.227.15:4873 , piece_id=KWBCMITJF25CZGDDHP5W265BBOLVLYMBQXGXCUGNNCNQR42FATMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.417366221 (520h49m3.288449266s) [noise=true , address=145.79.227.36:4518 , piece_id=NWDO2VJURYQTGSJR2Z5AWOGW6A7YYKX6XCMIFRFQ3NCOPXZ4DO3A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.417896100 (520h49m3.288979145s) [noise=true , address=145.79.227.41:4087 , piece_id=T2ENSAMVCNLVYDP2EHG6EXAESGJ4GAMUDVB56SIEKBJS2KZORX7A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.419488491 (520h49m3.290571526s) [noise=true , address=145.79.227.16:4910 , piece_id=GUJIJECGUQXBZ6U73LNY7TLRRPSUQRYSWNNZ5HM7SL7IK6LMI2CA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.421143082 (520h49m3.292226117s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.422762676 (520h49m3.293845731s) [noise=true , address=145.79.227.9:4366 , piece_id=AE7D36RORINH25A6ADLVCEJVPZ7ZPHO33Z6M3UR5XLRO77WDLXHA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.424467642 (520h49m3.295550687s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.427029470 (520h49m3.298112525s) [noise=true , address=45.91.116.7:4009 , piece_id=DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.435009705 (520h49m3.30609275s) [noise=true , address=145.79.227.48:4302 , piece_id=KX2IHHW6U72TCGNJN7OWMLJJLGUDRILVZW5B3LQQEBHZPFGLEZRQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.436529579 (520h49m3.307612644s) [noise=true , address=145.79.227.42:4123 , piece_id=KZZVK6VO2MP7K6TEMWRBPZL7VFXVHRB3URT3ELA4OULN47UQH6LA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:05.137399108 (520h49m5.008482153s) [upload=456900]: Next returned nil segment Dec 27 08:18:05.137415839 (520h49m5.008498874s) [upload=456900]: Waiting for error group managing segments... Dec 27 08:18:05.424338223 (520h49m5.295421258s) [noise=true , address=145.79.227.17:4971 , piece_id=C5JBC4TQVGRB743HMZMSNKTWLMDFR5GDTGMCKVUUZSSBH2CEGKMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.431048259 (520h49m5.302131294s) [noise=true , address=145.79.227.21:4474 , piece_id=4MLFNK3F5XUY6ZTHEZS2DZFLQGMHE2LQCDBHVPMX2XRHVOT7JYMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.433232056 (520h49m5.304315091s) [noise=true , address=145.79.227.27:4689 , piece_id=KJGZYISMDIJ7MQKJVGTVLXPSY765B4J4BSUAIBLO3YXI55GGT6PA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.436520538 (520h49m5.307603583s) [noise=true , address=145.79.227.41:4087 , piece_id=T2ENSAMVCNLVYDP2EHG6EXAESGJ4GAMUDVB56SIEKBJS2KZORX7A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.439087706 (520h49m5.310170741s) [noise=true , address=145.79.227.36:4518 , piece_id=NWDO2VJURYQTGSJR2Z5AWOGW6A7YYKX6XCMIFRFQ3NCOPXZ4DO3A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.439514829 (520h49m5.310597874s) [noise=true , address=145.79.227.50:4375 , piece_id=JIWLDRLSKO6U7NZWOG4HYXDM4QRBDLWOH3KR6BGKMQT23RTUERDQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.449397306 (520h49m5.320480352s) [noise=true , address=145.79.227.15:4873 , piece_id=KWBCMITJF25CZGDDHP5W265BBOLVLYMBQXGXCUGNNCNQR42FATMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.460860082 (520h49m5.331943117s) [noise=true , address=145.79.227.48:4302 , piece_id=KX2IHHW6U72TCGNJN7OWMLJJLGUDRILVZW5B3LQQEBHZPFGLEZRQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.463065833 (520h49m5.334148878s) [noise=true , address=145.79.227.9:4366 , piece_id=AE7D36RORINH25A6ADLVCEJVPZ7ZPHO33Z6M3UR5XLRO77WDLXHA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.463235656 (520h49m5.334318691s) [noise=true , address=145.79.227.42:4123 , piece_id=KZZVK6VO2MP7K6TEMWRBPZL7VFXVHRB3URT3ELA4OULN47UQH6LA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.467664649 (520h49m5.338747684s) [noise=true , address=145.79.227.16:4910 , piece_id=GUJIJECGUQXBZ6U73LNY7TLRRPSUQRYSWNNZ5HM7SL7IK6LMI2CA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.644554571 (520h49m5.515637606s) [noise=true , address=45.91.116.7:4009 , piece_id=DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.667329857 (520h49m5.538412902s) [noise=true , address=77.90.31.11:4554 , piece_id=EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.667355425 (520h49m5.53843847s) [seg_pos={0 0}, upload=456900]: Segment reached optimal threshold of 13 pieces. Dec 27 08:18:05.667588589 (520h49m5.538671645s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667606284 (520h49m5.538689329s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667666938 (520h49m5.538749993s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667676658 (520h49m5.538759703s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667832364 (520h49m5.538915409s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667841861 (520h49m5.538924907s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667866168 (520h49m5.538949203s) [seg_pos={0 0}, upload=456900]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:18:05.667958023 (520h49m5.539041058s) [upload=456900]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:18:05.667965738 (520h49m5.539048773s) [upload=456900]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:18:05.962502107 (520h49m5.833585142s) [upload=456900]: Done uploading segments... Dec 27 08:18:05.962522546 (520h49m5.833605591s) [upload=456900]: Upload finished. err: <nil> Duration: 5.904421001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........X......X.....X.X....X.X......X......X.XX..X....X..X.] duration: 5.720287755s trace: https://jaeger.dev.storj.io/trace/1a2df166b5f1cebe FAIL (2025-12-27 08:19:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A) context deadline exceeded ======LOGS====== Dec 27 08:18:03.171091985 (520h49m3.04217503s) [upload=456900]: Starting upload Dec 27 08:18:03.171179261 (520h49m3.042262307s) [upload=456900]: Done starting upload Dec 27 08:18:03.171312204 (520h49m3.042395239s) [upload=456900]: Uploading segments... Dec 27 08:18:03.171468322 (520h49m3.042551357s) [upload=456900]: Got next segment. Inline: false Dec 27 08:18:03.171506925 (520h49m3.04258997s) [upload=456900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:18:03.171514399 (520h49m3.042597434s) [upload=456900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:18:03.414596757 (520h49m3.285679812s) [seg_pos={0 0}, upload=456900]: Begin upload segment... Dec 27 08:18:03.414669295 (520h49m3.28575234s) [seg_pos={0 0}, upload=456900]: Done begin upload segment. Dec 27 08:18:03.414681088 (520h49m3.285764133s) [noise=true , address=145.79.227.17:4971 , piece_id=C5JBC4TQVGRB743HMZMSNKTWLMDFR5GDTGMCKVUUZSSBH2CEGKMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.414711426 (520h49m3.285794471s) [noise=true , address=145.79.227.27:4689 , piece_id=KJGZYISMDIJ7MQKJVGTVLXPSY765B4J4BSUAIBLO3YXI55GGT6PA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.415309094 (520h49m3.286392139s) [noise=true , address=145.79.227.21:4474 , piece_id=4MLFNK3F5XUY6ZTHEZS2DZFLQGMHE2LQCDBHVPMX2XRHVOT7JYMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.415698856 (520h49m3.286781901s) [noise=true , address=77.90.31.11:4554 , piece_id=EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416086915 (520h49m3.28716996s) [noise=true , address=145.79.227.50:4375 , piece_id=JIWLDRLSKO6U7NZWOG4HYXDM4QRBDLWOH3KR6BGKMQT23RTUERDQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416494040 (520h49m3.287577085s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416913338 (520h49m3.287996384s) [noise=true , address=145.79.227.15:4873 , piece_id=KWBCMITJF25CZGDDHP5W265BBOLVLYMBQXGXCUGNNCNQR42FATMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.417366221 (520h49m3.288449266s) [noise=true , address=145.79.227.36:4518 , piece_id=NWDO2VJURYQTGSJR2Z5AWOGW6A7YYKX6XCMIFRFQ3NCOPXZ4DO3A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.417896100 (520h49m3.288979145s) [noise=true , address=145.79.227.41:4087 , piece_id=T2ENSAMVCNLVYDP2EHG6EXAESGJ4GAMUDVB56SIEKBJS2KZORX7A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.419488491 (520h49m3.290571526s) [noise=true , address=145.79.227.16:4910 , piece_id=GUJIJECGUQXBZ6U73LNY7TLRRPSUQRYSWNNZ5HM7SL7IK6LMI2CA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.421143082 (520h49m3.292226117s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.422762676 (520h49m3.293845731s) [noise=true , address=145.79.227.9:4366 , piece_id=AE7D36RORINH25A6ADLVCEJVPZ7ZPHO33Z6M3UR5XLRO77WDLXHA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.424467642 (520h49m3.295550687s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.427029470 (520h49m3.298112525s) [noise=true , address=45.91.116.7:4009 , piece_id=DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.435009705 (520h49m3.30609275s) [noise=true , address=145.79.227.48:4302 , piece_id=KX2IHHW6U72TCGNJN7OWMLJJLGUDRILVZW5B3LQQEBHZPFGLEZRQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.436529579 (520h49m3.307612644s) [noise=true , address=145.79.227.42:4123 , piece_id=KZZVK6VO2MP7K6TEMWRBPZL7VFXVHRB3URT3ELA4OULN47UQH6LA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:05.137399108 (520h49m5.008482153s) [upload=456900]: Next returned nil segment Dec 27 08:18:05.137415839 (520h49m5.008498874s) [upload=456900]: Waiting for error group managing segments... Dec 27 08:18:05.424338223 (520h49m5.295421258s) [noise=true , address=145.79.227.17:4971 , piece_id=C5JBC4TQVGRB743HMZMSNKTWLMDFR5GDTGMCKVUUZSSBH2CEGKMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.431048259 (520h49m5.302131294s) [noise=true , address=145.79.227.21:4474 , piece_id=4MLFNK3F5XUY6ZTHEZS2DZFLQGMHE2LQCDBHVPMX2XRHVOT7JYMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.433232056 (520h49m5.304315091s) [noise=true , address=145.79.227.27:4689 , piece_id=KJGZYISMDIJ7MQKJVGTVLXPSY765B4J4BSUAIBLO3YXI55GGT6PA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.436520538 (520h49m5.307603583s) [noise=true , address=145.79.227.41:4087 , piece_id=T2ENSAMVCNLVYDP2EHG6EXAESGJ4GAMUDVB56SIEKBJS2KZORX7A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.439087706 (520h49m5.310170741s) [noise=true , address=145.79.227.36:4518 , piece_id=NWDO2VJURYQTGSJR2Z5AWOGW6A7YYKX6XCMIFRFQ3NCOPXZ4DO3A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.439514829 (520h49m5.310597874s) [noise=true , address=145.79.227.50:4375 , piece_id=JIWLDRLSKO6U7NZWOG4HYXDM4QRBDLWOH3KR6BGKMQT23RTUERDQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.449397306 (520h49m5.320480352s) [noise=true , address=145.79.227.15:4873 , piece_id=KWBCMITJF25CZGDDHP5W265BBOLVLYMBQXGXCUGNNCNQR42FATMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.460860082 (520h49m5.331943117s) [noise=true , address=145.79.227.48:4302 , piece_id=KX2IHHW6U72TCGNJN7OWMLJJLGUDRILVZW5B3LQQEBHZPFGLEZRQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.463065833 (520h49m5.334148878s) [noise=true , address=145.79.227.9:4366 , piece_id=AE7D36RORINH25A6ADLVCEJVPZ7ZPHO33Z6M3UR5XLRO77WDLXHA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.463235656 (520h49m5.334318691s) [noise=true , address=145.79.227.42:4123 , piece_id=KZZVK6VO2MP7K6TEMWRBPZL7VFXVHRB3URT3ELA4OULN47UQH6LA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.467664649 (520h49m5.338747684s) [noise=true , address=145.79.227.16:4910 , piece_id=GUJIJECGUQXBZ6U73LNY7TLRRPSUQRYSWNNZ5HM7SL7IK6LMI2CA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.644554571 (520h49m5.515637606s) [noise=true , address=45.91.116.7:4009 , piece_id=DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.667329857 (520h49m5.538412902s) [noise=true , address=77.90.31.11:4554 , piece_id=EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.667355425 (520h49m5.53843847s) [seg_pos={0 0}, upload=456900]: Segment reached optimal threshold of 13 pieces. Dec 27 08:18:05.667588589 (520h49m5.538671645s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667606284 (520h49m5.538689329s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667666938 (520h49m5.538749993s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667676658 (520h49m5.538759703s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667832364 (520h49m5.538915409s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667841861 (520h49m5.538924907s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667866168 (520h49m5.538949203s) [seg_pos={0 0}, upload=456900]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:18:05.667958023 (520h49m5.539041058s) [upload=456900]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:18:05.667965738 (520h49m5.539048773s) [upload=456900]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:18:05.962502107 (520h49m5.833585142s) [upload=456900]: Done uploading segments... Dec 27 08:18:05.962522546 (520h49m5.833605591s) [upload=456900]: Upload finished. err: <nil> Duration: 5.720317581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........X......X.....X.X....X.X......X......X.XX..X....X..X] duration: 30.003145159s trace: https://jaeger.dev.storj.io/trace/1a2df166b5f1cebe FAIL (2025-12-27 08:18:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A) context deadline exceeded ======LOGS====== Dec 27 08:18:03.171091985 (520h49m3.04217503s) [upload=456900]: Starting upload Dec 27 08:18:03.171179261 (520h49m3.042262307s) [upload=456900]: Done starting upload Dec 27 08:18:03.171312204 (520h49m3.042395239s) [upload=456900]: Uploading segments... Dec 27 08:18:03.171468322 (520h49m3.042551357s) [upload=456900]: Got next segment. Inline: false Dec 27 08:18:03.171506925 (520h49m3.04258997s) [upload=456900]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:18:03.171514399 (520h49m3.042597434s) [upload=456900]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:18:03.414596757 (520h49m3.285679812s) [seg_pos={0 0}, upload=456900]: Begin upload segment... Dec 27 08:18:03.414669295 (520h49m3.28575234s) [seg_pos={0 0}, upload=456900]: Done begin upload segment. Dec 27 08:18:03.414681088 (520h49m3.285764133s) [noise=true , address=145.79.227.17:4971 , piece_id=C5JBC4TQVGRB743HMZMSNKTWLMDFR5GDTGMCKVUUZSSBH2CEGKMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.414711426 (520h49m3.285794471s) [noise=true , address=145.79.227.27:4689 , piece_id=KJGZYISMDIJ7MQKJVGTVLXPSY765B4J4BSUAIBLO3YXI55GGT6PA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.415309094 (520h49m3.286392139s) [noise=true , address=145.79.227.21:4474 , piece_id=4MLFNK3F5XUY6ZTHEZS2DZFLQGMHE2LQCDBHVPMX2XRHVOT7JYMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.415698856 (520h49m3.286781901s) [noise=true , address=77.90.31.11:4554 , piece_id=EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416086915 (520h49m3.28716996s) [noise=true , address=145.79.227.50:4375 , piece_id=JIWLDRLSKO6U7NZWOG4HYXDM4QRBDLWOH3KR6BGKMQT23RTUERDQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416494040 (520h49m3.287577085s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.416913338 (520h49m3.287996384s) [noise=true , address=145.79.227.15:4873 , piece_id=KWBCMITJF25CZGDDHP5W265BBOLVLYMBQXGXCUGNNCNQR42FATMQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.417366221 (520h49m3.288449266s) [noise=true , address=145.79.227.36:4518 , piece_id=NWDO2VJURYQTGSJR2Z5AWOGW6A7YYKX6XCMIFRFQ3NCOPXZ4DO3A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.417896100 (520h49m3.288979145s) [noise=true , address=145.79.227.41:4087 , piece_id=T2ENSAMVCNLVYDP2EHG6EXAESGJ4GAMUDVB56SIEKBJS2KZORX7A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.419488491 (520h49m3.290571526s) [noise=true , address=145.79.227.16:4910 , piece_id=GUJIJECGUQXBZ6U73LNY7TLRRPSUQRYSWNNZ5HM7SL7IK6LMI2CA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.421143082 (520h49m3.292226117s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.422762676 (520h49m3.293845731s) [noise=true , address=145.79.227.9:4366 , piece_id=AE7D36RORINH25A6ADLVCEJVPZ7ZPHO33Z6M3UR5XLRO77WDLXHA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.424467642 (520h49m3.295550687s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.427029470 (520h49m3.298112525s) [noise=true , address=45.91.116.7:4009 , piece_id=DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.435009705 (520h49m3.30609275s) [noise=true , address=145.79.227.48:4302 , piece_id=KX2IHHW6U72TCGNJN7OWMLJJLGUDRILVZW5B3LQQEBHZPFGLEZRQ, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:03.436529579 (520h49m3.307612644s) [noise=true , address=145.79.227.42:4123 , piece_id=KZZVK6VO2MP7K6TEMWRBPZL7VFXVHRB3URT3ELA4OULN47UQH6LA, seg_pos={0 0}, upload=456900]: Uploading piece... Dec 27 08:18:05.137399108 (520h49m5.008482153s) [upload=456900]: Next returned nil segment Dec 27 08:18:05.137415839 (520h49m5.008498874s) [upload=456900]: Waiting for error group managing segments... Dec 27 08:18:05.424338223 (520h49m5.295421258s) [noise=true , address=145.79.227.17:4971 , piece_id=C5JBC4TQVGRB743HMZMSNKTWLMDFR5GDTGMCKVUUZSSBH2CEGKMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.431048259 (520h49m5.302131294s) [noise=true , address=145.79.227.21:4474 , piece_id=4MLFNK3F5XUY6ZTHEZS2DZFLQGMHE2LQCDBHVPMX2XRHVOT7JYMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.433232056 (520h49m5.304315091s) [noise=true , address=145.79.227.27:4689 , piece_id=KJGZYISMDIJ7MQKJVGTVLXPSY765B4J4BSUAIBLO3YXI55GGT6PA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.436520538 (520h49m5.307603583s) [noise=true , address=145.79.227.41:4087 , piece_id=T2ENSAMVCNLVYDP2EHG6EXAESGJ4GAMUDVB56SIEKBJS2KZORX7A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.439087706 (520h49m5.310170741s) [noise=true , address=145.79.227.36:4518 , piece_id=NWDO2VJURYQTGSJR2Z5AWOGW6A7YYKX6XCMIFRFQ3NCOPXZ4DO3A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.439514829 (520h49m5.310597874s) [noise=true , address=145.79.227.50:4375 , piece_id=JIWLDRLSKO6U7NZWOG4HYXDM4QRBDLWOH3KR6BGKMQT23RTUERDQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.449397306 (520h49m5.320480352s) [noise=true , address=145.79.227.15:4873 , piece_id=KWBCMITJF25CZGDDHP5W265BBOLVLYMBQXGXCUGNNCNQR42FATMQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.460860082 (520h49m5.331943117s) [noise=true , address=145.79.227.48:4302 , piece_id=KX2IHHW6U72TCGNJN7OWMLJJLGUDRILVZW5B3LQQEBHZPFGLEZRQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.463065833 (520h49m5.334148878s) [noise=true , address=145.79.227.9:4366 , piece_id=AE7D36RORINH25A6ADLVCEJVPZ7ZPHO33Z6M3UR5XLRO77WDLXHA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.463235656 (520h49m5.334318691s) [noise=true , address=145.79.227.42:4123 , piece_id=KZZVK6VO2MP7K6TEMWRBPZL7VFXVHRB3URT3ELA4OULN47UQH6LA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.467664649 (520h49m5.338747684s) [noise=true , address=145.79.227.16:4910 , piece_id=GUJIJECGUQXBZ6U73LNY7TLRRPSUQRYSWNNZ5HM7SL7IK6LMI2CA, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.644554571 (520h49m5.515637606s) [noise=true , address=45.91.116.7:4009 , piece_id=DO552BEHIUSLK6ODIK76GLKX3BSCFXBXB43NAVVOR5Q5SI6NLN2A, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.667329857 (520h49m5.538412902s) [noise=true , address=77.90.31.11:4554 , piece_id=EYDARASQADYAVYCXZTEKC5UCJUI5ZPLTDK25LUIWTIL24LRBE6DQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: <nil> Dec 27 08:18:05.667355425 (520h49m5.53843847s) [seg_pos={0 0}, upload=456900]: Segment reached optimal threshold of 13 pieces. Dec 27 08:18:05.667588589 (520h49m5.538671645s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667606284 (520h49m5.538689329s) [noise=true , address=45.91.116.10:4082 , piece_id=YETR3DRIGOMSI7MBQQWDBXXGNGS2UVV2X2QAXIGJPX4R7CD4JRFQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667666938 (520h49m5.538749993s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667676658 (520h49m5.538759703s) [noise=true , address=77.90.31.10:4520 , piece_id=U6SPBH2MBITKISVBZUSEEQOH5CNNQ6N5XYVE47DZRPHB2NUCXRDQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667832364 (520h49m5.538915409s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 08:18:05.667841861 (520h49m5.538924907s) [noise=true , address=45.91.116.5:4299 , piece_id=ZNBPJTOGVYZ7JTK4PNEP5RWSENO7WVQLJF2E24R6BGED7Q343DIQ, seg_pos={0 0}, upload=456900]: Long tail cancellation due to optimal threshold Dec 27 08:18:05.667866168 (520h49m5.538949203s) [seg_pos={0 0}, upload=456900]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:18:05.667958023 (520h49m5.539041058s) [upload=456900]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:18:05.667965738 (520h49m5.539048773s) [upload=456900]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:18:05.962502107 (520h49m5.833585142s) [upload=456900]: Done uploading segments... Dec 27 08:18:05.962522546 (520h49m5.833605591s) [upload=456900]: Upload finished. err: <nil> Duration: 30.003168724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X......X.....X.X....X.X......X......X.XX..X....X..] duration: 5.742903359s trace: https://jaeger.dev.storj.io/trace/335cd4362309f158 FAIL (2025-12-27 08:17:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ) context deadline exceeded ======LOGS====== Dec 27 08:16:09.277225913 (520h47m9.148308948s) [upload=456872]: Starting upload Dec 27 08:16:09.277315865 (520h47m9.1483989s) [upload=456872]: Done starting upload Dec 27 08:16:09.277388583 (520h47m9.148471628s) [upload=456872]: Uploading segments... Dec 27 08:16:09.277657374 (520h47m9.148740429s) [upload=456872]: Got next segment. Inline: false Dec 27 08:16:09.277723781 (520h47m9.148806826s) [upload=456872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:16:09.277730795 (520h47m9.14881383s) [upload=456872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:16:09.521393648 (520h47m9.392476693s) [seg_pos={0 0}, upload=456872]: Begin upload segment... Dec 27 08:16:09.521443703 (520h47m9.392526748s) [seg_pos={0 0}, upload=456872]: Done begin upload segment. Dec 27 08:16:09.521491915 (520h47m9.39257495s) [noise=true , address=145.79.227.38:4977 , piece_id=KBMKSN5KZSFXPNRUMP5T3PEHY3GZKZQ3GQFMGIBR27EXF7YM6EKA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.521497185 (520h47m9.39258022s) [noise=true , address=145.79.227.9:4366 , piece_id=45W7Z5DWC3VRK77SVV2PUJENC246VSLZH33IWVXZ6DCEYBMQGYDA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.522027604 (520h47m9.39311064s) [noise=true , address=145.79.227.16:4910 , piece_id=W6G7YLDK34CK2WTMKA6GVM7LHYIAKWZBIC3BRY7ZWUCPVZUWOJCQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.522535061 (520h47m9.393618106s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.523602533 (520h47m9.394685568s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.523987707 (520h47m9.395070742s) [noise=true , address=45.91.116.14:4192 , piece_id=HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.524257410 (520h47m9.395340445s) [noise=true , address=145.79.227.4:4185 , piece_id=JBB3TNY4MQL3YCTNHACE7U5X4HTOWVEFI242V2634ZXK7XDM24SA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.524603099 (520h47m9.395686144s) [noise=true , address=145.79.227.37:4566 , piece_id=HEQTCBLKXNGT2QIXVTNVZDPQTYMK7EXNHCWC5A5TVQ7GNV7FBBBA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.525163566 (520h47m9.396246601s) [noise=true , address=45.91.116.9:4263 , piece_id=66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.526313105 (520h47m9.39739615s) [noise=true , address=145.79.227.41:4085 , piece_id=BV33YIJYI4ZNLOJTO6GNKSCGXKGNDEIHOXDBCH6O5Q5NKDVURGPA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.527547916 (520h47m9.398630961s) [noise=true , address=145.79.227.17:4970 , piece_id=GVNEU72RIISWWZQN6XCL7X5LC2WWKFFS2AI2TKAXYEK7FTQHB5AQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.528623734 (520h47m9.399706769s) [noise=true , address=145.79.227.28:4726 , piece_id=LCFNLFJHLOCGQ6SZPLZNKFDGFXHBGIQMK7NCBUXF5P2MMCZJ5CBQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.534117755 (520h47m9.40520079s) [noise=true , address=145.79.227.39:4014 , piece_id=3VYMEBZSAKVZE4S2OAYKZW6CTCOG2BEASAZJTVX77TJCFZPGUSJA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.536190051 (520h47m9.407273086s) [noise=true , address=77.90.31.18:4808 , piece_id=VQGVWXUF72LUEB7YDLDVIFRDAO7AJEK37DU7D5RVSEIJ3BMGRQEQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.537415605 (520h47m9.40849864s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.537896751 (520h47m9.408979796s) [noise=true , address=145.79.227.46:4602 , piece_id=OJXXBXCGEM3DLQ4UA4HHTJFP4EETGAQTJL7DNUY6WVZVLJJHPPZQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:11.244488124 (520h47m11.115571169s) [upload=456872]: Next returned nil segment Dec 27 08:16:11.244505187 (520h47m11.115588232s) [upload=456872]: Waiting for error group managing segments... Dec 27 08:16:11.529631169 (520h47m11.400714214s) [noise=true , address=145.79.227.9:4366 , piece_id=45W7Z5DWC3VRK77SVV2PUJENC246VSLZH33IWVXZ6DCEYBMQGYDA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.532231981 (520h47m11.403315016s) [noise=true , address=145.79.227.17:4970 , piece_id=GVNEU72RIISWWZQN6XCL7X5LC2WWKFFS2AI2TKAXYEK7FTQHB5AQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.532609801 (520h47m11.403692836s) [noise=true , address=145.79.227.37:4566 , piece_id=HEQTCBLKXNGT2QIXVTNVZDPQTYMK7EXNHCWC5A5TVQ7GNV7FBBBA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.540368163 (520h47m11.411451208s) [noise=true , address=145.79.227.28:4726 , piece_id=LCFNLFJHLOCGQ6SZPLZNKFDGFXHBGIQMK7NCBUXF5P2MMCZJ5CBQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.546069058 (520h47m11.417152113s) [noise=true , address=145.79.227.4:4185 , piece_id=JBB3TNY4MQL3YCTNHACE7U5X4HTOWVEFI242V2634ZXK7XDM24SA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.552734259 (520h47m11.423817304s) [noise=true , address=145.79.227.46:4602 , piece_id=OJXXBXCGEM3DLQ4UA4HHTJFP4EETGAQTJL7DNUY6WVZVLJJHPPZQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.554419558 (520h47m11.425502603s) [noise=true , address=145.79.227.16:4910 , piece_id=W6G7YLDK34CK2WTMKA6GVM7LHYIAKWZBIC3BRY7ZWUCPVZUWOJCQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.557415463 (520h47m11.428498498s) [noise=true , address=145.79.227.39:4014 , piece_id=3VYMEBZSAKVZE4S2OAYKZW6CTCOG2BEASAZJTVX77TJCFZPGUSJA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.568824806 (520h47m11.439907851s) [noise=true , address=145.79.227.41:4085 , piece_id=BV33YIJYI4ZNLOJTO6GNKSCGXKGNDEIHOXDBCH6O5Q5NKDVURGPA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.708346687 (520h47m11.579429732s) [noise=true , address=77.90.31.18:4808 , piece_id=VQGVWXUF72LUEB7YDLDVIFRDAO7AJEK37DU7D5RVSEIJ3BMGRQEQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.747320847 (520h47m11.618403892s) [noise=true , address=45.91.116.14:4192 , piece_id=HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.786739794 (520h47m11.657822839s) [noise=true , address=45.91.116.9:4263 , piece_id=66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.792717035 (520h47m11.66380008s) [noise=true , address=145.79.227.38:4977 , piece_id=KBMKSN5KZSFXPNRUMP5T3PEHY3GZKZQ3GQFMGIBR27EXF7YM6EKA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.792740099 (520h47m11.663823144s) [seg_pos={0 0}, upload=456872]: Segment reached optimal threshold of 13 pieces. Dec 27 08:16:11.793013901 (520h47m11.664096946s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection Dec 27 08:16:11.793024120 (520h47m11.664107165s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793177181 (520h47m11.664260226s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 08:16:11.793227137 (520h47m11.664310172s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 08:16:11.793237797 (520h47m11.664320842s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793252795 (520h47m11.66433584s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793344259 (520h47m11.664427304s) [seg_pos={0 0}, upload=456872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:16:11.793472875 (520h47m11.66455592s) [upload=456872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:16:11.793479066 (520h47m11.664562111s) [upload=456872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:16:12.089882681 (520h47m11.960965716s) [upload=456872]: Done uploading segments... Dec 27 08:16:12.089902038 (520h47m11.960985073s) [upload=456872]: Upload finished. err: <nil> Duration: 5.742931472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..........X......X.....X.X....X.X......X......X.XX..X....X.] duration: 5.858545309s trace: https://jaeger.dev.storj.io/trace/335cd4362309f158 FAIL (2025-12-27 08:16:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ) context deadline exceeded ======LOGS====== Dec 27 08:16:09.277225913 (520h47m9.148308948s) [upload=456872]: Starting upload Dec 27 08:16:09.277315865 (520h47m9.1483989s) [upload=456872]: Done starting upload Dec 27 08:16:09.277388583 (520h47m9.148471628s) [upload=456872]: Uploading segments... Dec 27 08:16:09.277657374 (520h47m9.148740429s) [upload=456872]: Got next segment. Inline: false Dec 27 08:16:09.277723781 (520h47m9.148806826s) [upload=456872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:16:09.277730795 (520h47m9.14881383s) [upload=456872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:16:09.521393648 (520h47m9.392476693s) [seg_pos={0 0}, upload=456872]: Begin upload segment... Dec 27 08:16:09.521443703 (520h47m9.392526748s) [seg_pos={0 0}, upload=456872]: Done begin upload segment. Dec 27 08:16:09.521491915 (520h47m9.39257495s) [noise=true , address=145.79.227.38:4977 , piece_id=KBMKSN5KZSFXPNRUMP5T3PEHY3GZKZQ3GQFMGIBR27EXF7YM6EKA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.521497185 (520h47m9.39258022s) [noise=true , address=145.79.227.9:4366 , piece_id=45W7Z5DWC3VRK77SVV2PUJENC246VSLZH33IWVXZ6DCEYBMQGYDA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.522027604 (520h47m9.39311064s) [noise=true , address=145.79.227.16:4910 , piece_id=W6G7YLDK34CK2WTMKA6GVM7LHYIAKWZBIC3BRY7ZWUCPVZUWOJCQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.522535061 (520h47m9.393618106s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.523602533 (520h47m9.394685568s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.523987707 (520h47m9.395070742s) [noise=true , address=45.91.116.14:4192 , piece_id=HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.524257410 (520h47m9.395340445s) [noise=true , address=145.79.227.4:4185 , piece_id=JBB3TNY4MQL3YCTNHACE7U5X4HTOWVEFI242V2634ZXK7XDM24SA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.524603099 (520h47m9.395686144s) [noise=true , address=145.79.227.37:4566 , piece_id=HEQTCBLKXNGT2QIXVTNVZDPQTYMK7EXNHCWC5A5TVQ7GNV7FBBBA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.525163566 (520h47m9.396246601s) [noise=true , address=45.91.116.9:4263 , piece_id=66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.526313105 (520h47m9.39739615s) [noise=true , address=145.79.227.41:4085 , piece_id=BV33YIJYI4ZNLOJTO6GNKSCGXKGNDEIHOXDBCH6O5Q5NKDVURGPA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.527547916 (520h47m9.398630961s) [noise=true , address=145.79.227.17:4970 , piece_id=GVNEU72RIISWWZQN6XCL7X5LC2WWKFFS2AI2TKAXYEK7FTQHB5AQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.528623734 (520h47m9.399706769s) [noise=true , address=145.79.227.28:4726 , piece_id=LCFNLFJHLOCGQ6SZPLZNKFDGFXHBGIQMK7NCBUXF5P2MMCZJ5CBQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.534117755 (520h47m9.40520079s) [noise=true , address=145.79.227.39:4014 , piece_id=3VYMEBZSAKVZE4S2OAYKZW6CTCOG2BEASAZJTVX77TJCFZPGUSJA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.536190051 (520h47m9.407273086s) [noise=true , address=77.90.31.18:4808 , piece_id=VQGVWXUF72LUEB7YDLDVIFRDAO7AJEK37DU7D5RVSEIJ3BMGRQEQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.537415605 (520h47m9.40849864s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.537896751 (520h47m9.408979796s) [noise=true , address=145.79.227.46:4602 , piece_id=OJXXBXCGEM3DLQ4UA4HHTJFP4EETGAQTJL7DNUY6WVZVLJJHPPZQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:11.244488124 (520h47m11.115571169s) [upload=456872]: Next returned nil segment Dec 27 08:16:11.244505187 (520h47m11.115588232s) [upload=456872]: Waiting for error group managing segments... Dec 27 08:16:11.529631169 (520h47m11.400714214s) [noise=true , address=145.79.227.9:4366 , piece_id=45W7Z5DWC3VRK77SVV2PUJENC246VSLZH33IWVXZ6DCEYBMQGYDA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.532231981 (520h47m11.403315016s) [noise=true , address=145.79.227.17:4970 , piece_id=GVNEU72RIISWWZQN6XCL7X5LC2WWKFFS2AI2TKAXYEK7FTQHB5AQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.532609801 (520h47m11.403692836s) [noise=true , address=145.79.227.37:4566 , piece_id=HEQTCBLKXNGT2QIXVTNVZDPQTYMK7EXNHCWC5A5TVQ7GNV7FBBBA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.540368163 (520h47m11.411451208s) [noise=true , address=145.79.227.28:4726 , piece_id=LCFNLFJHLOCGQ6SZPLZNKFDGFXHBGIQMK7NCBUXF5P2MMCZJ5CBQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.546069058 (520h47m11.417152113s) [noise=true , address=145.79.227.4:4185 , piece_id=JBB3TNY4MQL3YCTNHACE7U5X4HTOWVEFI242V2634ZXK7XDM24SA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.552734259 (520h47m11.423817304s) [noise=true , address=145.79.227.46:4602 , piece_id=OJXXBXCGEM3DLQ4UA4HHTJFP4EETGAQTJL7DNUY6WVZVLJJHPPZQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.554419558 (520h47m11.425502603s) [noise=true , address=145.79.227.16:4910 , piece_id=W6G7YLDK34CK2WTMKA6GVM7LHYIAKWZBIC3BRY7ZWUCPVZUWOJCQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.557415463 (520h47m11.428498498s) [noise=true , address=145.79.227.39:4014 , piece_id=3VYMEBZSAKVZE4S2OAYKZW6CTCOG2BEASAZJTVX77TJCFZPGUSJA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.568824806 (520h47m11.439907851s) [noise=true , address=145.79.227.41:4085 , piece_id=BV33YIJYI4ZNLOJTO6GNKSCGXKGNDEIHOXDBCH6O5Q5NKDVURGPA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.708346687 (520h47m11.579429732s) [noise=true , address=77.90.31.18:4808 , piece_id=VQGVWXUF72LUEB7YDLDVIFRDAO7AJEK37DU7D5RVSEIJ3BMGRQEQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.747320847 (520h47m11.618403892s) [noise=true , address=45.91.116.14:4192 , piece_id=HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.786739794 (520h47m11.657822839s) [noise=true , address=45.91.116.9:4263 , piece_id=66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.792717035 (520h47m11.66380008s) [noise=true , address=145.79.227.38:4977 , piece_id=KBMKSN5KZSFXPNRUMP5T3PEHY3GZKZQ3GQFMGIBR27EXF7YM6EKA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.792740099 (520h47m11.663823144s) [seg_pos={0 0}, upload=456872]: Segment reached optimal threshold of 13 pieces. Dec 27 08:16:11.793013901 (520h47m11.664096946s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection Dec 27 08:16:11.793024120 (520h47m11.664107165s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793177181 (520h47m11.664260226s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 08:16:11.793227137 (520h47m11.664310172s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 08:16:11.793237797 (520h47m11.664320842s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793252795 (520h47m11.66433584s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793344259 (520h47m11.664427304s) [seg_pos={0 0}, upload=456872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:16:11.793472875 (520h47m11.66455592s) [upload=456872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:16:11.793479066 (520h47m11.664562111s) [upload=456872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:16:12.089882681 (520h47m11.960965716s) [upload=456872]: Done uploading segments... Dec 27 08:16:12.089902038 (520h47m11.960985073s) [upload=456872]: Upload finished. err: <nil> Duration: 5.858565306s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..........X......X.....X.X....X.X......X......X.XX..X....X] duration: 30.003203707s trace: https://jaeger.dev.storj.io/trace/335cd4362309f158 FAIL (2025-12-27 08:16:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ) context deadline exceeded ======LOGS====== Dec 27 08:16:09.277225913 (520h47m9.148308948s) [upload=456872]: Starting upload Dec 27 08:16:09.277315865 (520h47m9.1483989s) [upload=456872]: Done starting upload Dec 27 08:16:09.277388583 (520h47m9.148471628s) [upload=456872]: Uploading segments... Dec 27 08:16:09.277657374 (520h47m9.148740429s) [upload=456872]: Got next segment. Inline: false Dec 27 08:16:09.277723781 (520h47m9.148806826s) [upload=456872]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:16:09.277730795 (520h47m9.14881383s) [upload=456872]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:16:09.521393648 (520h47m9.392476693s) [seg_pos={0 0}, upload=456872]: Begin upload segment... Dec 27 08:16:09.521443703 (520h47m9.392526748s) [seg_pos={0 0}, upload=456872]: Done begin upload segment. Dec 27 08:16:09.521491915 (520h47m9.39257495s) [noise=true , address=145.79.227.38:4977 , piece_id=KBMKSN5KZSFXPNRUMP5T3PEHY3GZKZQ3GQFMGIBR27EXF7YM6EKA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.521497185 (520h47m9.39258022s) [noise=true , address=145.79.227.9:4366 , piece_id=45W7Z5DWC3VRK77SVV2PUJENC246VSLZH33IWVXZ6DCEYBMQGYDA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.522027604 (520h47m9.39311064s) [noise=true , address=145.79.227.16:4910 , piece_id=W6G7YLDK34CK2WTMKA6GVM7LHYIAKWZBIC3BRY7ZWUCPVZUWOJCQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.522535061 (520h47m9.393618106s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.523602533 (520h47m9.394685568s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.523987707 (520h47m9.395070742s) [noise=true , address=45.91.116.14:4192 , piece_id=HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.524257410 (520h47m9.395340445s) [noise=true , address=145.79.227.4:4185 , piece_id=JBB3TNY4MQL3YCTNHACE7U5X4HTOWVEFI242V2634ZXK7XDM24SA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.524603099 (520h47m9.395686144s) [noise=true , address=145.79.227.37:4566 , piece_id=HEQTCBLKXNGT2QIXVTNVZDPQTYMK7EXNHCWC5A5TVQ7GNV7FBBBA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.525163566 (520h47m9.396246601s) [noise=true , address=45.91.116.9:4263 , piece_id=66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.526313105 (520h47m9.39739615s) [noise=true , address=145.79.227.41:4085 , piece_id=BV33YIJYI4ZNLOJTO6GNKSCGXKGNDEIHOXDBCH6O5Q5NKDVURGPA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.527547916 (520h47m9.398630961s) [noise=true , address=145.79.227.17:4970 , piece_id=GVNEU72RIISWWZQN6XCL7X5LC2WWKFFS2AI2TKAXYEK7FTQHB5AQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.528623734 (520h47m9.399706769s) [noise=true , address=145.79.227.28:4726 , piece_id=LCFNLFJHLOCGQ6SZPLZNKFDGFXHBGIQMK7NCBUXF5P2MMCZJ5CBQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.534117755 (520h47m9.40520079s) [noise=true , address=145.79.227.39:4014 , piece_id=3VYMEBZSAKVZE4S2OAYKZW6CTCOG2BEASAZJTVX77TJCFZPGUSJA, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.536190051 (520h47m9.407273086s) [noise=true , address=77.90.31.18:4808 , piece_id=VQGVWXUF72LUEB7YDLDVIFRDAO7AJEK37DU7D5RVSEIJ3BMGRQEQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.537415605 (520h47m9.40849864s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:09.537896751 (520h47m9.408979796s) [noise=true , address=145.79.227.46:4602 , piece_id=OJXXBXCGEM3DLQ4UA4HHTJFP4EETGAQTJL7DNUY6WVZVLJJHPPZQ, seg_pos={0 0}, upload=456872]: Uploading piece... Dec 27 08:16:11.244488124 (520h47m11.115571169s) [upload=456872]: Next returned nil segment Dec 27 08:16:11.244505187 (520h47m11.115588232s) [upload=456872]: Waiting for error group managing segments... Dec 27 08:16:11.529631169 (520h47m11.400714214s) [noise=true , address=145.79.227.9:4366 , piece_id=45W7Z5DWC3VRK77SVV2PUJENC246VSLZH33IWVXZ6DCEYBMQGYDA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.532231981 (520h47m11.403315016s) [noise=true , address=145.79.227.17:4970 , piece_id=GVNEU72RIISWWZQN6XCL7X5LC2WWKFFS2AI2TKAXYEK7FTQHB5AQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.532609801 (520h47m11.403692836s) [noise=true , address=145.79.227.37:4566 , piece_id=HEQTCBLKXNGT2QIXVTNVZDPQTYMK7EXNHCWC5A5TVQ7GNV7FBBBA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.540368163 (520h47m11.411451208s) [noise=true , address=145.79.227.28:4726 , piece_id=LCFNLFJHLOCGQ6SZPLZNKFDGFXHBGIQMK7NCBUXF5P2MMCZJ5CBQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.546069058 (520h47m11.417152113s) [noise=true , address=145.79.227.4:4185 , piece_id=JBB3TNY4MQL3YCTNHACE7U5X4HTOWVEFI242V2634ZXK7XDM24SA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.552734259 (520h47m11.423817304s) [noise=true , address=145.79.227.46:4602 , piece_id=OJXXBXCGEM3DLQ4UA4HHTJFP4EETGAQTJL7DNUY6WVZVLJJHPPZQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.554419558 (520h47m11.425502603s) [noise=true , address=145.79.227.16:4910 , piece_id=W6G7YLDK34CK2WTMKA6GVM7LHYIAKWZBIC3BRY7ZWUCPVZUWOJCQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.557415463 (520h47m11.428498498s) [noise=true , address=145.79.227.39:4014 , piece_id=3VYMEBZSAKVZE4S2OAYKZW6CTCOG2BEASAZJTVX77TJCFZPGUSJA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.568824806 (520h47m11.439907851s) [noise=true , address=145.79.227.41:4085 , piece_id=BV33YIJYI4ZNLOJTO6GNKSCGXKGNDEIHOXDBCH6O5Q5NKDVURGPA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.708346687 (520h47m11.579429732s) [noise=true , address=77.90.31.18:4808 , piece_id=VQGVWXUF72LUEB7YDLDVIFRDAO7AJEK37DU7D5RVSEIJ3BMGRQEQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.747320847 (520h47m11.618403892s) [noise=true , address=45.91.116.14:4192 , piece_id=HWO2LVGBU2DG7RUHEA6HDML7YR2LMNEG2HNKCPKUU75RCRVDJQNQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.786739794 (520h47m11.657822839s) [noise=true , address=45.91.116.9:4263 , piece_id=66IEFF6AN4QJ7V7VPN7HAZR76RL32JGQT6UCQDCTXKFMZ25PF3MQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.792717035 (520h47m11.66380008s) [noise=true , address=145.79.227.38:4977 , piece_id=KBMKSN5KZSFXPNRUMP5T3PEHY3GZKZQ3GQFMGIBR27EXF7YM6EKA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: <nil> Dec 27 08:16:11.792740099 (520h47m11.663823144s) [seg_pos={0 0}, upload=456872]: Segment reached optimal threshold of 13 pieces. Dec 27 08:16:11.793013901 (520h47m11.664096946s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53536->77.90.31.11:4556: use of closed network connection Dec 27 08:16:11.793024120 (520h47m11.664107165s) [noise=true , address=77.90.31.11:4556 , piece_id=3Y6FLLTRKHYO7UMCDUZDRPH66OD5QNGLRCNNWQIYU6XAYJKJDGRA, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793177181 (520h47m11.664260226s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 08:16:11.793227137 (520h47m11.664310172s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 08:16:11.793237797 (520h47m11.664320842s) [noise=true , address=45.91.116.13:4156 , piece_id=JSS4B326DULUOD227VNERH7DZIGSADZYVG7KXG3MIMAJDZAYPXMA, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793252795 (520h47m11.66433584s) [noise=true , address=45.91.116.11:4120 , piece_id=SEEKNA4P7ZP5NJ372WLM6EJUSJZACZVSA2RCCVDPN4EH7WIIR7GQ, seg_pos={0 0}, upload=456872]: Long tail cancellation due to optimal threshold Dec 27 08:16:11.793344259 (520h47m11.664427304s) [seg_pos={0 0}, upload=456872]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:16:11.793472875 (520h47m11.66455592s) [upload=456872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:16:11.793479066 (520h47m11.664562111s) [upload=456872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:16:12.089882681 (520h47m11.960965716s) [upload=456872]: Done uploading segments... Dec 27 08:16:12.089902038 (520h47m11.960985073s) [upload=456872]: Upload finished. err: <nil> Duration: 30.003225158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X..........X......X.....X.X....X.X......X......X.XX..X....] duration: 6.024042627s trace: https://jaeger.dev.storj.io/trace/966ff92be4ad74e FAIL (2025-12-27 08:15:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ) context deadline exceeded ======LOGS====== Dec 27 08:11:20.067929052 (520h42m19.939012097s) [upload=456805]: Starting upload Dec 27 08:11:20.067998304 (520h42m19.939081349s) [upload=456805]: Done starting upload Dec 27 08:11:20.068105067 (520h42m19.939188102s) [upload=456805]: Uploading segments... Dec 27 08:11:20.068305780 (520h42m19.939388825s) [upload=456805]: Got next segment. Inline: false Dec 27 08:11:20.068365272 (520h42m19.939448317s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:11:20.068371224 (520h42m19.939454259s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:11:20.314187668 (520h42m20.185270713s) [seg_pos={0 0}, upload=456805]: Begin upload segment... Dec 27 08:11:20.314249396 (520h42m20.185332431s) [seg_pos={0 0}, upload=456805]: Done begin upload segment. Dec 27 08:11:20.314326683 (520h42m20.185409718s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314388971 (520h42m20.185472026s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314936062 (520h42m20.186019108s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315214793 (520h42m20.186297848s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315696571 (520h42m20.186779606s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.316068179 (520h42m20.187151214s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317604103 (520h42m20.188687138s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317948679 (520h42m20.189031714s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.318446547 (520h42m20.189529582s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.319259264 (520h42m20.190342299s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.320653459 (520h42m20.191736494s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322239799 (520h42m20.193322834s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322802731 (520h42m20.193885766s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.326084349 (520h42m20.197167384s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327810817 (520h42m20.198893852s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327989237 (520h42m20.199072272s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:22.143720301 (520h42m22.014803336s) [upload=456805]: Next returned nil segment Dec 27 08:11:22.143734928 (520h42m22.014817963s) [upload=456805]: Waiting for error group managing segments... Dec 27 08:11:22.444784270 (520h42m22.315867305s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.449473378 (520h42m22.320556413s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.453741424 (520h42m22.32482446s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.458816178 (520h42m22.329899223s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.463110915 (520h42m22.334193961s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470016905 (520h42m22.34109994s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470726246 (520h42m22.341809281s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.475005213 (520h42m22.346088248s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.577778221 (520h42m22.448861256s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.639458735 (520h42m22.51054177s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.677124944 (520h42m22.548207989s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.141463060 (520h42m23.012546095s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289103921 (520h42m23.160186966s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289134760 (520h42m23.160217805s) [seg_pos={0 0}, upload=456805]: Segment reached optimal threshold of 13 pieces. Dec 27 08:11:23.289348416 (520h42m23.160431461s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289359187 (520h42m23.160442232s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289682301 (520h42m23.160765357s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289690056 (520h42m23.160773101s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289899505 (520h42m23.16098256s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289908122 (520h42m23.160991177s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289931957 (520h42m23.161015002s) [seg_pos={0 0}, upload=456805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:11:23.290014815 (520h42m23.16109786s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:11:23.290021838 (520h42m23.161104883s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:11:23.585194991 (520h42m23.456278026s) [upload=456805]: Done uploading segments... Dec 27 08:11:23.585214869 (520h42m23.456297904s) [upload=456805]: Upload finished. err: <nil> Duration: 6.024071562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X..........X......X.....X.X....X.X......X......X.XX..X...] duration: 6.213106663s trace: https://jaeger.dev.storj.io/trace/966ff92be4ad74e FAIL (2025-12-27 08:14:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ) context deadline exceeded ======LOGS====== Dec 27 08:11:20.067929052 (520h42m19.939012097s) [upload=456805]: Starting upload Dec 27 08:11:20.067998304 (520h42m19.939081349s) [upload=456805]: Done starting upload Dec 27 08:11:20.068105067 (520h42m19.939188102s) [upload=456805]: Uploading segments... Dec 27 08:11:20.068305780 (520h42m19.939388825s) [upload=456805]: Got next segment. Inline: false Dec 27 08:11:20.068365272 (520h42m19.939448317s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:11:20.068371224 (520h42m19.939454259s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:11:20.314187668 (520h42m20.185270713s) [seg_pos={0 0}, upload=456805]: Begin upload segment... Dec 27 08:11:20.314249396 (520h42m20.185332431s) [seg_pos={0 0}, upload=456805]: Done begin upload segment. Dec 27 08:11:20.314326683 (520h42m20.185409718s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314388971 (520h42m20.185472026s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314936062 (520h42m20.186019108s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315214793 (520h42m20.186297848s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315696571 (520h42m20.186779606s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.316068179 (520h42m20.187151214s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317604103 (520h42m20.188687138s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317948679 (520h42m20.189031714s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.318446547 (520h42m20.189529582s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.319259264 (520h42m20.190342299s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.320653459 (520h42m20.191736494s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322239799 (520h42m20.193322834s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322802731 (520h42m20.193885766s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.326084349 (520h42m20.197167384s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327810817 (520h42m20.198893852s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327989237 (520h42m20.199072272s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:22.143720301 (520h42m22.014803336s) [upload=456805]: Next returned nil segment Dec 27 08:11:22.143734928 (520h42m22.014817963s) [upload=456805]: Waiting for error group managing segments... Dec 27 08:11:22.444784270 (520h42m22.315867305s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.449473378 (520h42m22.320556413s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.453741424 (520h42m22.32482446s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.458816178 (520h42m22.329899223s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.463110915 (520h42m22.334193961s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470016905 (520h42m22.34109994s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470726246 (520h42m22.341809281s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.475005213 (520h42m22.346088248s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.577778221 (520h42m22.448861256s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.639458735 (520h42m22.51054177s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.677124944 (520h42m22.548207989s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.141463060 (520h42m23.012546095s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289103921 (520h42m23.160186966s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289134760 (520h42m23.160217805s) [seg_pos={0 0}, upload=456805]: Segment reached optimal threshold of 13 pieces. Dec 27 08:11:23.289348416 (520h42m23.160431461s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289359187 (520h42m23.160442232s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289682301 (520h42m23.160765357s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289690056 (520h42m23.160773101s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289899505 (520h42m23.16098256s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289908122 (520h42m23.160991177s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289931957 (520h42m23.161015002s) [seg_pos={0 0}, upload=456805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:11:23.290014815 (520h42m23.16109786s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:11:23.290021838 (520h42m23.161104883s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:11:23.585194991 (520h42m23.456278026s) [upload=456805]: Done uploading segments... Dec 27 08:11:23.585214869 (520h42m23.456297904s) [upload=456805]: Upload finished. err: <nil> Duration: 6.213151737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..X..........X......X.....X.X....X.X......X......X.XX..X..] duration: 6.323690733s trace: https://jaeger.dev.storj.io/trace/966ff92be4ad74e FAIL (2025-12-27 08:13:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ) context deadline exceeded ======LOGS====== Dec 27 08:11:20.067929052 (520h42m19.939012097s) [upload=456805]: Starting upload Dec 27 08:11:20.067998304 (520h42m19.939081349s) [upload=456805]: Done starting upload Dec 27 08:11:20.068105067 (520h42m19.939188102s) [upload=456805]: Uploading segments... Dec 27 08:11:20.068305780 (520h42m19.939388825s) [upload=456805]: Got next segment. Inline: false Dec 27 08:11:20.068365272 (520h42m19.939448317s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:11:20.068371224 (520h42m19.939454259s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:11:20.314187668 (520h42m20.185270713s) [seg_pos={0 0}, upload=456805]: Begin upload segment... Dec 27 08:11:20.314249396 (520h42m20.185332431s) [seg_pos={0 0}, upload=456805]: Done begin upload segment. Dec 27 08:11:20.314326683 (520h42m20.185409718s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314388971 (520h42m20.185472026s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314936062 (520h42m20.186019108s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315214793 (520h42m20.186297848s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315696571 (520h42m20.186779606s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.316068179 (520h42m20.187151214s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317604103 (520h42m20.188687138s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317948679 (520h42m20.189031714s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.318446547 (520h42m20.189529582s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.319259264 (520h42m20.190342299s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.320653459 (520h42m20.191736494s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322239799 (520h42m20.193322834s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322802731 (520h42m20.193885766s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.326084349 (520h42m20.197167384s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327810817 (520h42m20.198893852s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327989237 (520h42m20.199072272s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:22.143720301 (520h42m22.014803336s) [upload=456805]: Next returned nil segment Dec 27 08:11:22.143734928 (520h42m22.014817963s) [upload=456805]: Waiting for error group managing segments... Dec 27 08:11:22.444784270 (520h42m22.315867305s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.449473378 (520h42m22.320556413s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.453741424 (520h42m22.32482446s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.458816178 (520h42m22.329899223s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.463110915 (520h42m22.334193961s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470016905 (520h42m22.34109994s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470726246 (520h42m22.341809281s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.475005213 (520h42m22.346088248s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.577778221 (520h42m22.448861256s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.639458735 (520h42m22.51054177s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.677124944 (520h42m22.548207989s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.141463060 (520h42m23.012546095s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289103921 (520h42m23.160186966s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289134760 (520h42m23.160217805s) [seg_pos={0 0}, upload=456805]: Segment reached optimal threshold of 13 pieces. Dec 27 08:11:23.289348416 (520h42m23.160431461s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289359187 (520h42m23.160442232s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289682301 (520h42m23.160765357s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289690056 (520h42m23.160773101s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289899505 (520h42m23.16098256s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289908122 (520h42m23.160991177s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289931957 (520h42m23.161015002s) [seg_pos={0 0}, upload=456805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:11:23.290014815 (520h42m23.16109786s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:11:23.290021838 (520h42m23.161104883s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:11:23.585194991 (520h42m23.456278026s) [upload=456805]: Done uploading segments... Dec 27 08:11:23.585214869 (520h42m23.456297904s) [upload=456805]: Upload finished. err: <nil> Duration: 6.323717844s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..X..........X......X.....X.X....X.X......X......X.XX..X.] duration: 6.959147882s trace: https://jaeger.dev.storj.io/trace/966ff92be4ad74e FAIL (2025-12-27 08:12:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ) context deadline exceeded ======LOGS====== Dec 27 08:11:20.067929052 (520h42m19.939012097s) [upload=456805]: Starting upload Dec 27 08:11:20.067998304 (520h42m19.939081349s) [upload=456805]: Done starting upload Dec 27 08:11:20.068105067 (520h42m19.939188102s) [upload=456805]: Uploading segments... Dec 27 08:11:20.068305780 (520h42m19.939388825s) [upload=456805]: Got next segment. Inline: false Dec 27 08:11:20.068365272 (520h42m19.939448317s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:11:20.068371224 (520h42m19.939454259s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:11:20.314187668 (520h42m20.185270713s) [seg_pos={0 0}, upload=456805]: Begin upload segment... Dec 27 08:11:20.314249396 (520h42m20.185332431s) [seg_pos={0 0}, upload=456805]: Done begin upload segment. Dec 27 08:11:20.314326683 (520h42m20.185409718s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314388971 (520h42m20.185472026s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314936062 (520h42m20.186019108s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315214793 (520h42m20.186297848s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315696571 (520h42m20.186779606s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.316068179 (520h42m20.187151214s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317604103 (520h42m20.188687138s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317948679 (520h42m20.189031714s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.318446547 (520h42m20.189529582s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.319259264 (520h42m20.190342299s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.320653459 (520h42m20.191736494s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322239799 (520h42m20.193322834s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322802731 (520h42m20.193885766s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.326084349 (520h42m20.197167384s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327810817 (520h42m20.198893852s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327989237 (520h42m20.199072272s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:22.143720301 (520h42m22.014803336s) [upload=456805]: Next returned nil segment Dec 27 08:11:22.143734928 (520h42m22.014817963s) [upload=456805]: Waiting for error group managing segments... Dec 27 08:11:22.444784270 (520h42m22.315867305s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.449473378 (520h42m22.320556413s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.453741424 (520h42m22.32482446s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.458816178 (520h42m22.329899223s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.463110915 (520h42m22.334193961s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470016905 (520h42m22.34109994s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470726246 (520h42m22.341809281s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.475005213 (520h42m22.346088248s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.577778221 (520h42m22.448861256s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.639458735 (520h42m22.51054177s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.677124944 (520h42m22.548207989s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.141463060 (520h42m23.012546095s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289103921 (520h42m23.160186966s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289134760 (520h42m23.160217805s) [seg_pos={0 0}, upload=456805]: Segment reached optimal threshold of 13 pieces. Dec 27 08:11:23.289348416 (520h42m23.160431461s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289359187 (520h42m23.160442232s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289682301 (520h42m23.160765357s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289690056 (520h42m23.160773101s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289899505 (520h42m23.16098256s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289908122 (520h42m23.160991177s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289931957 (520h42m23.161015002s) [seg_pos={0 0}, upload=456805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:11:23.290014815 (520h42m23.16109786s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:11:23.290021838 (520h42m23.161104883s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:11:23.585194991 (520h42m23.456278026s) [upload=456805]: Done uploading segments... Dec 27 08:11:23.585214869 (520h42m23.456297904s) [upload=456805]: Upload finished. err: <nil> Duration: 6.959190432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X..........X......X.....X.X....X.X......X......X.XX..X] duration: 30.002398335s trace: https://jaeger.dev.storj.io/trace/966ff92be4ad74e FAIL (2025-12-27 08:11:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ) context deadline exceeded ======LOGS====== Dec 27 08:11:20.067929052 (520h42m19.939012097s) [upload=456805]: Starting upload Dec 27 08:11:20.067998304 (520h42m19.939081349s) [upload=456805]: Done starting upload Dec 27 08:11:20.068105067 (520h42m19.939188102s) [upload=456805]: Uploading segments... Dec 27 08:11:20.068305780 (520h42m19.939388825s) [upload=456805]: Got next segment. Inline: false Dec 27 08:11:20.068365272 (520h42m19.939448317s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:11:20.068371224 (520h42m19.939454259s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:11:20.314187668 (520h42m20.185270713s) [seg_pos={0 0}, upload=456805]: Begin upload segment... Dec 27 08:11:20.314249396 (520h42m20.185332431s) [seg_pos={0 0}, upload=456805]: Done begin upload segment. Dec 27 08:11:20.314326683 (520h42m20.185409718s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314388971 (520h42m20.185472026s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.314936062 (520h42m20.186019108s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315214793 (520h42m20.186297848s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.315696571 (520h42m20.186779606s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.316068179 (520h42m20.187151214s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317604103 (520h42m20.188687138s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.317948679 (520h42m20.189031714s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.318446547 (520h42m20.189529582s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.319259264 (520h42m20.190342299s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.320653459 (520h42m20.191736494s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322239799 (520h42m20.193322834s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.322802731 (520h42m20.193885766s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.326084349 (520h42m20.197167384s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327810817 (520h42m20.198893852s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:20.327989237 (520h42m20.199072272s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Uploading piece... Dec 27 08:11:22.143720301 (520h42m22.014803336s) [upload=456805]: Next returned nil segment Dec 27 08:11:22.143734928 (520h42m22.014817963s) [upload=456805]: Waiting for error group managing segments... Dec 27 08:11:22.444784270 (520h42m22.315867305s) [noise=true , address=145.79.227.13:4802 , piece_id=IMSVPADRMEBZ5BV747S7CKG5D5ZFHH6E4VMELWOVLOOW5SUGVAAQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.449473378 (520h42m22.320556413s) [noise=true , address=145.79.227.5:4221 , piece_id=CZM2OC4GQRR3NGQY2SMM76K3CVK55AQ4EOYKK445QDCZZJ2MWIHQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.453741424 (520h42m22.32482446s) [noise=true , address=145.79.227.37:4565 , piece_id=4KOBSMV4KNYGQHK4DMN5NCB5GXCDI4D6ZSZMSQPBXTZLQYQL4SPA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.458816178 (520h42m22.329899223s) [noise=true , address=145.79.227.4:4185 , piece_id=LLW7BQV2II3XKCFAPCLHJZQQYZPL2QJ726EMX7KBRRX3ENIG4UNQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.463110915 (520h42m22.334193961s) [noise=true , address=145.79.227.48:4301 , piece_id=DLLXWB5RPIS5AV7FUULUADXYHETDDQJ2KUMFVRGWIDANQOSAZSLQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470016905 (520h42m22.34109994s) [noise=true , address=145.79.227.16:4909 , piece_id=GPLDIMA6YVSZGROCGQF6SENHWKNEX3EQIA5YGLGXGIAWUNQPQIIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.470726246 (520h42m22.341809281s) [noise=true , address=145.79.227.36:4519 , piece_id=QAFZ36BFT4UL3SIYFM2LAYYTPIDIE6UER5BOQKVDAK7CPI4NTIHA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.475005213 (520h42m22.346088248s) [noise=true , address=145.79.227.28:4726 , piece_id=4K2URRRL4KDGJWRV4XBCYGHWC5REOXVPDC2AHPTR6UNB4QS45USA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.577778221 (520h42m22.448861256s) [noise=true , address=45.91.116.13:4155 , piece_id=K75U7LDDP4NN4SMYHL5VJK5VOFCMIJZ4TQVCE4BU22STQR27V67Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.639458735 (520h42m22.51054177s) [noise=true , address=77.90.31.15:4700 , piece_id=BBHXEG6O7ZWZBVCDSKPVTREJQ3UIITRRVG2JEKDGWORBEFG6HRXQ, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:22.677124944 (520h42m22.548207989s) [noise=true , address=45.91.116.5:4300 , piece_id=D4YEHWZCWVZ2O6GXHCJKJJ2HEKVX6FOK6LNPCSRDYDHJ6276DLMA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.141463060 (520h42m23.012546095s) [noise=true , address=145.79.227.27:4689 , piece_id=DH47G4KYHCP6U3EM6HITYTBYHQH4PPFCUDFYRQL33MEYJ7IJ6Z7Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289103921 (520h42m23.160186966s) [noise=true , address=45.91.116.7:4010 , piece_id=YDX3SU44B5S2QEGVQD4QDDLBUAAJS3MV4HDFKFGU2742CN7JISIA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: <nil> Dec 27 08:11:23.289134760 (520h42m23.160217805s) [seg_pos={0 0}, upload=456805]: Segment reached optimal threshold of 13 pieces. Dec 27 08:11:23.289348416 (520h42m23.160431461s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289359187 (520h42m23.160442232s) [noise=true , address=45.91.116.4:4939 , piece_id=RFO7DYQJOLYZ5EF7QLTMUSBC2J72SIZ7DQ3WJK7J35LF7J6WDI2Q, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289682301 (520h42m23.160765357s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289690056 (520h42m23.160773101s) [noise=true , address=45.91.116.14:4191 , piece_id=UXYLN6MI4YQGVFOIYO2IVWDO5BL5O7VDWUNOZCSHUJ4NCMMEALXA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289899505 (520h42m23.16098256s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 27 08:11:23.289908122 (520h42m23.160991177s) [noise=true , address=45.91.116.8:4046 , piece_id=NGS4QCNONB4KR6D2DVSBRDZ6LC3QASUL3WWKUNWQ5GL6NDG7U4XA, seg_pos={0 0}, upload=456805]: Long tail cancellation due to optimal threshold Dec 27 08:11:23.289931957 (520h42m23.161015002s) [seg_pos={0 0}, upload=456805]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:11:23.290014815 (520h42m23.16109786s) [upload=456805]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:11:23.290021838 (520h42m23.161104883s) [upload=456805]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:11:23.585194991 (520h42m23.456278026s) [upload=456805]: Done uploading segments... Dec 27 08:11:23.585214869 (520h42m23.456297904s) [upload=456805]: Upload finished. err: <nil> Duration: 30.002426748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X..X..........X......X.....X.X....X.X......X......X.XX..] duration: 11.167768899s trace: https://jaeger.dev.storj.io/trace/723d4494c898a905 FAIL (2025-12-27 08:11:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA) context deadline exceeded ======LOGS====== Dec 27 08:09:58.524584826 (520h40m58.395667871s) [upload=456784]: Starting upload Dec 27 08:09:58.524690156 (520h40m58.395773201s) [upload=456784]: Done starting upload Dec 27 08:09:58.524755050 (520h40m58.395838085s) [upload=456784]: Uploading segments... Dec 27 08:09:58.524930925 (520h40m58.39601397s) [upload=456784]: Got next segment. Inline: false Dec 27 08:09:58.524988814 (520h40m58.396071859s) [upload=456784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:09:58.524995909 (520h40m58.396078954s) [upload=456784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:09:58.778315659 (520h40m58.649398704s) [seg_pos={0 0}, upload=456784]: Begin upload segment... Dec 27 08:09:58.778371695 (520h40m58.64945473s) [seg_pos={0 0}, upload=456784]: Done begin upload segment. Dec 27 08:09:58.778444243 (520h40m58.649527278s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.778456608 (520h40m58.649539643s) [noise=true , address=77.90.31.19:4842 , piece_id=RYIEGQTQ5TQMFURRGRWPWVEBNLDFBRJEVQX2VCFOYYLBVMYDVQRA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.778962100 (520h40m58.650045145s) [noise=true , address=145.79.227.35:4483 , piece_id=P4U2HHCNIL37GHEMHMLRMGFBDG6SAJ7AK5H4XK55LRMS6MFFILKQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.779419811 (520h40m58.650502846s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.779777102 (520h40m58.650860137s) [noise=true , address=145.79.227.41:4086 , piece_id=DREW4GGO5LVT2NMJ5K5JVGLAUWOYXN2HFLRANQ4QPOXRLSJU4FMQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780084056 (520h40m58.651167091s) [noise=true , address=145.79.227.11:4732 , piece_id=IAXOKYK44JQ2P3QH26NL4DLQIEQ3JD2EI6SFFJ6VQM4GGGYUWS5A, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780394487 (520h40m58.651477522s) [noise=true , address=145.79.227.27:4689 , piece_id=YZUALO2PJY4I7LUPHNWDRWYU2KSA6NFAD43TLNQ4LCUZ5WCV25XQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780722833 (520h40m58.651805868s) [noise=true , address=145.79.227.22:4510 , piece_id=VAGZDBL7YQOKNO5VDXGNLDKQJ4FPADXKNJKH2TEPBKIWJBAKXWWQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.781071166 (520h40m58.652154201s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.781412084 (520h40m58.652495119s) [noise=true , address=145.79.227.48:4302 , piece_id=D2MBYUMPFCKWSA4XSANHCDFJVMCUJACKIDTIGQH2MSCYGK3X32KQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.782886432 (520h40m58.653969467s) [noise=true , address=45.91.116.8:4046 , piece_id=XFK6TSTROJREZNKQZR6KQ7LVOETKMEV6F4JKDGZRANQSPDMJ5EWQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.784480277 (520h40m58.655563322s) [noise=true , address=145.79.227.36:4517 , piece_id=JEN7J6GAYBECCYYLULIJYPGKCGLBUVWK5M3WXKBKEK6ZZMWNIQCQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.784756553 (520h40m58.655839588s) [noise=true , address=45.91.116.7:4009 , piece_id=NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.786154805 (520h40m58.65723785s) [noise=true , address=45.91.116.9:4264 , piece_id=JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.786691246 (520h40m58.657774281s) [noise=true , address=45.91.116.6:4976 , piece_id=7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.789574867 (520h40m58.660657902s) [noise=true , address=145.79.227.50:4374 , piece_id=QZDLYRINFLGWWHUNASH5F6Z247NTRTHP7DTVPCV7SRYTSIA4JBVA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:10:00.407479614 (520h41m0.278562659s) [upload=456784]: Next returned nil segment Dec 27 08:10:00.407519981 (520h41m0.278603026s) [upload=456784]: Waiting for error group managing segments... Dec 27 08:10:00.790915747 (520h41m0.661998792s) [noise=true , address=145.79.227.48:4302 , piece_id=D2MBYUMPFCKWSA4XSANHCDFJVMCUJACKIDTIGQH2MSCYGK3X32KQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.791109055 (520h41m0.66219209s) [noise=true , address=145.79.227.36:4517 , piece_id=JEN7J6GAYBECCYYLULIJYPGKCGLBUVWK5M3WXKBKEK6ZZMWNIQCQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.796729597 (520h41m0.667812642s) [noise=true , address=145.79.227.11:4732 , piece_id=IAXOKYK44JQ2P3QH26NL4DLQIEQ3JD2EI6SFFJ6VQM4GGGYUWS5A, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.799882039 (520h41m0.670965084s) [noise=true , address=145.79.227.50:4374 , piece_id=QZDLYRINFLGWWHUNASH5F6Z247NTRTHP7DTVPCV7SRYTSIA4JBVA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.800423250 (520h41m0.671506295s) [noise=true , address=145.79.227.35:4483 , piece_id=P4U2HHCNIL37GHEMHMLRMGFBDG6SAJ7AK5H4XK55LRMS6MFFILKQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.803832982 (520h41m0.674916027s) [noise=true , address=145.79.227.27:4689 , piece_id=YZUALO2PJY4I7LUPHNWDRWYU2KSA6NFAD43TLNQ4LCUZ5WCV25XQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.807223799 (520h41m0.678306834s) [noise=true , address=145.79.227.41:4086 , piece_id=DREW4GGO5LVT2NMJ5K5JVGLAUWOYXN2HFLRANQ4QPOXRLSJU4FMQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.814914001 (520h41m0.685997047s) [noise=true , address=145.79.227.22:4510 , piece_id=VAGZDBL7YQOKNO5VDXGNLDKQJ4FPADXKNJKH2TEPBKIWJBAKXWWQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.035553015 (520h41m0.90663605s) [noise=true , address=45.91.116.8:4046 , piece_id=XFK6TSTROJREZNKQZR6KQ7LVOETKMEV6F4JKDGZRANQSPDMJ5EWQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.086757432 (520h41m0.957840477s) [noise=true , address=45.91.116.9:4264 , piece_id=JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.160183030 (520h41m1.031266075s) [noise=true , address=45.91.116.7:4009 , piece_id=NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.354121524 (520h41m1.225204579s) [noise=true , address=77.90.31.19:4842 , piece_id=RYIEGQTQ5TQMFURRGRWPWVEBNLDFBRJEVQX2VCFOYYLBVMYDVQRA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.490780985 (520h41m1.36186404s) [noise=true , address=45.91.116.6:4976 , piece_id=7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.490811483 (520h41m1.361894538s) [seg_pos={0 0}, upload=456784]: Segment reached optimal threshold of 13 pieces. Dec 27 08:10:01.491066708 (520h41m1.362149763s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection Dec 27 08:10:01.491077949 (520h41m1.362161015s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492117509 (520h41m1.363200564s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 08:10:01.492130203 (520h41m1.363213248s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492570412 (520h41m1.363653457s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 08:10:01.492583506 (520h41m1.363666551s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492610788 (520h41m1.363693833s) [seg_pos={0 0}, upload=456784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:10:01.492825266 (520h41m1.363908312s) [upload=456784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:10:01.492834765 (520h41m1.36391781s) [upload=456784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:10:01.780103147 (520h41m1.651186182s) [upload=456784]: Done uploading segments... Dec 27 08:10:01.780133134 (520h41m1.651216179s) [upload=456784]: Upload finished. err: <nil> Duration: 11.167826478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..X..........X......X.....X.X....X.X......X......X.XX.] duration: 5.693249602s trace: https://jaeger.dev.storj.io/trace/723d4494c898a905 FAIL (2025-12-27 08:10:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA) context deadline exceeded ======LOGS====== Dec 27 08:09:58.524584826 (520h40m58.395667871s) [upload=456784]: Starting upload Dec 27 08:09:58.524690156 (520h40m58.395773201s) [upload=456784]: Done starting upload Dec 27 08:09:58.524755050 (520h40m58.395838085s) [upload=456784]: Uploading segments... Dec 27 08:09:58.524930925 (520h40m58.39601397s) [upload=456784]: Got next segment. Inline: false Dec 27 08:09:58.524988814 (520h40m58.396071859s) [upload=456784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:09:58.524995909 (520h40m58.396078954s) [upload=456784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:09:58.778315659 (520h40m58.649398704s) [seg_pos={0 0}, upload=456784]: Begin upload segment... Dec 27 08:09:58.778371695 (520h40m58.64945473s) [seg_pos={0 0}, upload=456784]: Done begin upload segment. Dec 27 08:09:58.778444243 (520h40m58.649527278s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.778456608 (520h40m58.649539643s) [noise=true , address=77.90.31.19:4842 , piece_id=RYIEGQTQ5TQMFURRGRWPWVEBNLDFBRJEVQX2VCFOYYLBVMYDVQRA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.778962100 (520h40m58.650045145s) [noise=true , address=145.79.227.35:4483 , piece_id=P4U2HHCNIL37GHEMHMLRMGFBDG6SAJ7AK5H4XK55LRMS6MFFILKQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.779419811 (520h40m58.650502846s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.779777102 (520h40m58.650860137s) [noise=true , address=145.79.227.41:4086 , piece_id=DREW4GGO5LVT2NMJ5K5JVGLAUWOYXN2HFLRANQ4QPOXRLSJU4FMQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780084056 (520h40m58.651167091s) [noise=true , address=145.79.227.11:4732 , piece_id=IAXOKYK44JQ2P3QH26NL4DLQIEQ3JD2EI6SFFJ6VQM4GGGYUWS5A, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780394487 (520h40m58.651477522s) [noise=true , address=145.79.227.27:4689 , piece_id=YZUALO2PJY4I7LUPHNWDRWYU2KSA6NFAD43TLNQ4LCUZ5WCV25XQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780722833 (520h40m58.651805868s) [noise=true , address=145.79.227.22:4510 , piece_id=VAGZDBL7YQOKNO5VDXGNLDKQJ4FPADXKNJKH2TEPBKIWJBAKXWWQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.781071166 (520h40m58.652154201s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.781412084 (520h40m58.652495119s) [noise=true , address=145.79.227.48:4302 , piece_id=D2MBYUMPFCKWSA4XSANHCDFJVMCUJACKIDTIGQH2MSCYGK3X32KQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.782886432 (520h40m58.653969467s) [noise=true , address=45.91.116.8:4046 , piece_id=XFK6TSTROJREZNKQZR6KQ7LVOETKMEV6F4JKDGZRANQSPDMJ5EWQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.784480277 (520h40m58.655563322s) [noise=true , address=145.79.227.36:4517 , piece_id=JEN7J6GAYBECCYYLULIJYPGKCGLBUVWK5M3WXKBKEK6ZZMWNIQCQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.784756553 (520h40m58.655839588s) [noise=true , address=45.91.116.7:4009 , piece_id=NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.786154805 (520h40m58.65723785s) [noise=true , address=45.91.116.9:4264 , piece_id=JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.786691246 (520h40m58.657774281s) [noise=true , address=45.91.116.6:4976 , piece_id=7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.789574867 (520h40m58.660657902s) [noise=true , address=145.79.227.50:4374 , piece_id=QZDLYRINFLGWWHUNASH5F6Z247NTRTHP7DTVPCV7SRYTSIA4JBVA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:10:00.407479614 (520h41m0.278562659s) [upload=456784]: Next returned nil segment Dec 27 08:10:00.407519981 (520h41m0.278603026s) [upload=456784]: Waiting for error group managing segments... Dec 27 08:10:00.790915747 (520h41m0.661998792s) [noise=true , address=145.79.227.48:4302 , piece_id=D2MBYUMPFCKWSA4XSANHCDFJVMCUJACKIDTIGQH2MSCYGK3X32KQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.791109055 (520h41m0.66219209s) [noise=true , address=145.79.227.36:4517 , piece_id=JEN7J6GAYBECCYYLULIJYPGKCGLBUVWK5M3WXKBKEK6ZZMWNIQCQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.796729597 (520h41m0.667812642s) [noise=true , address=145.79.227.11:4732 , piece_id=IAXOKYK44JQ2P3QH26NL4DLQIEQ3JD2EI6SFFJ6VQM4GGGYUWS5A, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.799882039 (520h41m0.670965084s) [noise=true , address=145.79.227.50:4374 , piece_id=QZDLYRINFLGWWHUNASH5F6Z247NTRTHP7DTVPCV7SRYTSIA4JBVA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.800423250 (520h41m0.671506295s) [noise=true , address=145.79.227.35:4483 , piece_id=P4U2HHCNIL37GHEMHMLRMGFBDG6SAJ7AK5H4XK55LRMS6MFFILKQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.803832982 (520h41m0.674916027s) [noise=true , address=145.79.227.27:4689 , piece_id=YZUALO2PJY4I7LUPHNWDRWYU2KSA6NFAD43TLNQ4LCUZ5WCV25XQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.807223799 (520h41m0.678306834s) [noise=true , address=145.79.227.41:4086 , piece_id=DREW4GGO5LVT2NMJ5K5JVGLAUWOYXN2HFLRANQ4QPOXRLSJU4FMQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.814914001 (520h41m0.685997047s) [noise=true , address=145.79.227.22:4510 , piece_id=VAGZDBL7YQOKNO5VDXGNLDKQJ4FPADXKNJKH2TEPBKIWJBAKXWWQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.035553015 (520h41m0.90663605s) [noise=true , address=45.91.116.8:4046 , piece_id=XFK6TSTROJREZNKQZR6KQ7LVOETKMEV6F4JKDGZRANQSPDMJ5EWQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.086757432 (520h41m0.957840477s) [noise=true , address=45.91.116.9:4264 , piece_id=JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.160183030 (520h41m1.031266075s) [noise=true , address=45.91.116.7:4009 , piece_id=NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.354121524 (520h41m1.225204579s) [noise=true , address=77.90.31.19:4842 , piece_id=RYIEGQTQ5TQMFURRGRWPWVEBNLDFBRJEVQX2VCFOYYLBVMYDVQRA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.490780985 (520h41m1.36186404s) [noise=true , address=45.91.116.6:4976 , piece_id=7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.490811483 (520h41m1.361894538s) [seg_pos={0 0}, upload=456784]: Segment reached optimal threshold of 13 pieces. Dec 27 08:10:01.491066708 (520h41m1.362149763s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection Dec 27 08:10:01.491077949 (520h41m1.362161015s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492117509 (520h41m1.363200564s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 08:10:01.492130203 (520h41m1.363213248s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492570412 (520h41m1.363653457s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 08:10:01.492583506 (520h41m1.363666551s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492610788 (520h41m1.363693833s) [seg_pos={0 0}, upload=456784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:10:01.492825266 (520h41m1.363908312s) [upload=456784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:10:01.492834765 (520h41m1.36391781s) [upload=456784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:10:01.780103147 (520h41m1.651186182s) [upload=456784]: Done uploading segments... Dec 27 08:10:01.780133134 (520h41m1.651216179s) [upload=456784]: Upload finished. err: <nil> Duration: 5.693279119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X..X..........X......X.....X.X....X.X......X......X.XX] duration: 30.002360915s trace: https://jaeger.dev.storj.io/trace/723d4494c898a905 FAIL (2025-12-27 08:10:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA) context deadline exceeded ======LOGS====== Dec 27 08:09:58.524584826 (520h40m58.395667871s) [upload=456784]: Starting upload Dec 27 08:09:58.524690156 (520h40m58.395773201s) [upload=456784]: Done starting upload Dec 27 08:09:58.524755050 (520h40m58.395838085s) [upload=456784]: Uploading segments... Dec 27 08:09:58.524930925 (520h40m58.39601397s) [upload=456784]: Got next segment. Inline: false Dec 27 08:09:58.524988814 (520h40m58.396071859s) [upload=456784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:09:58.524995909 (520h40m58.396078954s) [upload=456784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:09:58.778315659 (520h40m58.649398704s) [seg_pos={0 0}, upload=456784]: Begin upload segment... Dec 27 08:09:58.778371695 (520h40m58.64945473s) [seg_pos={0 0}, upload=456784]: Done begin upload segment. Dec 27 08:09:58.778444243 (520h40m58.649527278s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.778456608 (520h40m58.649539643s) [noise=true , address=77.90.31.19:4842 , piece_id=RYIEGQTQ5TQMFURRGRWPWVEBNLDFBRJEVQX2VCFOYYLBVMYDVQRA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.778962100 (520h40m58.650045145s) [noise=true , address=145.79.227.35:4483 , piece_id=P4U2HHCNIL37GHEMHMLRMGFBDG6SAJ7AK5H4XK55LRMS6MFFILKQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.779419811 (520h40m58.650502846s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.779777102 (520h40m58.650860137s) [noise=true , address=145.79.227.41:4086 , piece_id=DREW4GGO5LVT2NMJ5K5JVGLAUWOYXN2HFLRANQ4QPOXRLSJU4FMQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780084056 (520h40m58.651167091s) [noise=true , address=145.79.227.11:4732 , piece_id=IAXOKYK44JQ2P3QH26NL4DLQIEQ3JD2EI6SFFJ6VQM4GGGYUWS5A, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780394487 (520h40m58.651477522s) [noise=true , address=145.79.227.27:4689 , piece_id=YZUALO2PJY4I7LUPHNWDRWYU2KSA6NFAD43TLNQ4LCUZ5WCV25XQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.780722833 (520h40m58.651805868s) [noise=true , address=145.79.227.22:4510 , piece_id=VAGZDBL7YQOKNO5VDXGNLDKQJ4FPADXKNJKH2TEPBKIWJBAKXWWQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.781071166 (520h40m58.652154201s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.781412084 (520h40m58.652495119s) [noise=true , address=145.79.227.48:4302 , piece_id=D2MBYUMPFCKWSA4XSANHCDFJVMCUJACKIDTIGQH2MSCYGK3X32KQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.782886432 (520h40m58.653969467s) [noise=true , address=45.91.116.8:4046 , piece_id=XFK6TSTROJREZNKQZR6KQ7LVOETKMEV6F4JKDGZRANQSPDMJ5EWQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.784480277 (520h40m58.655563322s) [noise=true , address=145.79.227.36:4517 , piece_id=JEN7J6GAYBECCYYLULIJYPGKCGLBUVWK5M3WXKBKEK6ZZMWNIQCQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.784756553 (520h40m58.655839588s) [noise=true , address=45.91.116.7:4009 , piece_id=NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.786154805 (520h40m58.65723785s) [noise=true , address=45.91.116.9:4264 , piece_id=JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.786691246 (520h40m58.657774281s) [noise=true , address=45.91.116.6:4976 , piece_id=7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:09:58.789574867 (520h40m58.660657902s) [noise=true , address=145.79.227.50:4374 , piece_id=QZDLYRINFLGWWHUNASH5F6Z247NTRTHP7DTVPCV7SRYTSIA4JBVA, seg_pos={0 0}, upload=456784]: Uploading piece... Dec 27 08:10:00.407479614 (520h41m0.278562659s) [upload=456784]: Next returned nil segment Dec 27 08:10:00.407519981 (520h41m0.278603026s) [upload=456784]: Waiting for error group managing segments... Dec 27 08:10:00.790915747 (520h41m0.661998792s) [noise=true , address=145.79.227.48:4302 , piece_id=D2MBYUMPFCKWSA4XSANHCDFJVMCUJACKIDTIGQH2MSCYGK3X32KQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.791109055 (520h41m0.66219209s) [noise=true , address=145.79.227.36:4517 , piece_id=JEN7J6GAYBECCYYLULIJYPGKCGLBUVWK5M3WXKBKEK6ZZMWNIQCQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.796729597 (520h41m0.667812642s) [noise=true , address=145.79.227.11:4732 , piece_id=IAXOKYK44JQ2P3QH26NL4DLQIEQ3JD2EI6SFFJ6VQM4GGGYUWS5A, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.799882039 (520h41m0.670965084s) [noise=true , address=145.79.227.50:4374 , piece_id=QZDLYRINFLGWWHUNASH5F6Z247NTRTHP7DTVPCV7SRYTSIA4JBVA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.800423250 (520h41m0.671506295s) [noise=true , address=145.79.227.35:4483 , piece_id=P4U2HHCNIL37GHEMHMLRMGFBDG6SAJ7AK5H4XK55LRMS6MFFILKQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.803832982 (520h41m0.674916027s) [noise=true , address=145.79.227.27:4689 , piece_id=YZUALO2PJY4I7LUPHNWDRWYU2KSA6NFAD43TLNQ4LCUZ5WCV25XQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.807223799 (520h41m0.678306834s) [noise=true , address=145.79.227.41:4086 , piece_id=DREW4GGO5LVT2NMJ5K5JVGLAUWOYXN2HFLRANQ4QPOXRLSJU4FMQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:00.814914001 (520h41m0.685997047s) [noise=true , address=145.79.227.22:4510 , piece_id=VAGZDBL7YQOKNO5VDXGNLDKQJ4FPADXKNJKH2TEPBKIWJBAKXWWQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.035553015 (520h41m0.90663605s) [noise=true , address=45.91.116.8:4046 , piece_id=XFK6TSTROJREZNKQZR6KQ7LVOETKMEV6F4JKDGZRANQSPDMJ5EWQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.086757432 (520h41m0.957840477s) [noise=true , address=45.91.116.9:4264 , piece_id=JWG2SYGIANJQG6FDRYE5HG3PF2Q5O6DCP6NGJUUS46M6EWC5EWQQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.160183030 (520h41m1.031266075s) [noise=true , address=45.91.116.7:4009 , piece_id=NMWESQSYOOASMDSPLGJR4G57YWPX4Q35HSMPPFX55YJFA2AOUUOQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.354121524 (520h41m1.225204579s) [noise=true , address=77.90.31.19:4842 , piece_id=RYIEGQTQ5TQMFURRGRWPWVEBNLDFBRJEVQX2VCFOYYLBVMYDVQRA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.490780985 (520h41m1.36186404s) [noise=true , address=45.91.116.6:4976 , piece_id=7UVG6KFJ67WJATCA3FBMMWRLU4O2LT6GNA4ALGF27AQ6CDGMNOWA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: <nil> Dec 27 08:10:01.490811483 (520h41m1.361894538s) [seg_pos={0 0}, upload=456784]: Segment reached optimal threshold of 13 pieces. Dec 27 08:10:01.491066708 (520h41m1.362149763s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57860->45.91.116.14:4191: use of closed network connection Dec 27 08:10:01.491077949 (520h41m1.362161015s) [noise=true , address=45.91.116.14:4191 , piece_id=W54425IL2XJZDCMZW3R66UTTZSBOJYTDLF2DC47BGKT3UK2WID5A, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492117509 (520h41m1.363200564s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 08:10:01.492130203 (520h41m1.363213248s) [noise=true , address=45.91.116.10:4084 , piece_id=NZ5PBS3R3LCRKM5LIFFX6QHAFDDRKH3UEQH3TSVAQKQ2HSHQT4SA, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492570412 (520h41m1.363653457s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 08:10:01.492583506 (520h41m1.363666551s) [noise=true , address=45.91.116.5:4298 , piece_id=K7IO3NNFH5QJPJJTUQI6YLL2252XD6GU4HBVFIF4REZLNFS2PLGQ, seg_pos={0 0}, upload=456784]: Long tail cancellation due to optimal threshold Dec 27 08:10:01.492610788 (520h41m1.363693833s) [seg_pos={0 0}, upload=456784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:10:01.492825266 (520h41m1.363908312s) [upload=456784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:10:01.492834765 (520h41m1.36391781s) [upload=456784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:10:01.780103147 (520h41m1.651186182s) [upload=456784]: Done uploading segments... Dec 27 08:10:01.780133134 (520h41m1.651216179s) [upload=456784]: Upload finished. err: <nil> Duration: 30.002387356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......X..X..........X......X.....X.X....X.X......X......X.X] duration: 30.00333955s trace: https://jaeger.dev.storj.io/trace/4fc9add4b7948c95 FAIL (2025-12-27 08:09:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 08:09:26.122682055 (520h40m25.9937651s) [upload=456775]: Starting upload Dec 27 08:09:26.122816662 (520h40m25.993899707s) [upload=456775]: Done starting upload Dec 27 08:09:26.122931932 (520h40m25.994014977s) [upload=456775]: Uploading segments... Dec 27 08:09:26.123217686 (520h40m25.994300721s) [upload=456775]: Got next segment. Inline: false Dec 27 08:09:26.123270606 (520h40m25.994353641s) [upload=456775]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:09:26.123274924 (520h40m25.994357969s) [upload=456775]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:09:26.369649652 (520h40m26.240732687s) [seg_pos={0 0}, upload=456775]: Begin upload segment... Dec 27 08:09:26.370037119 (520h40m26.241120154s) [seg_pos={0 0}, upload=456775]: Done begin upload segment. Dec 27 08:09:26.370071314 (520h40m26.24115435s) [noise=true , address=145.79.227.15:4873 , piece_id=KMFBA4MYMF7LLBJ3REURD5VLSX6RD23WU4MHLX2KUNLPRI235ASQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.370116711 (520h40m26.241199756s) [noise=true , address=145.79.227.51:4409 , piece_id=SIHKY6TAUP6TWJ3XHQDPCIFE5ZNJQK7DS7NOEBQ33ELPMXL2RECQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.370752822 (520h40m26.241835867s) [noise=true , address=145.79.227.28:4726 , piece_id=VVDYZ7UUS2YLVCMWII7CST5ZQRVNYC5ADO36TVQHULXXSE4ZJUJQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.371324090 (520h40m26.242407125s) [noise=true , address=77.90.31.14:4663 , piece_id=QZKGYYSTSPQAVRKQHTW4G6CNYBZXSDVSADUKOKXLUN5EGWRMGKSA, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.372212621 (520h40m26.243295656s) [noise=true , address=145.79.227.50:4374 , piece_id=B5NAGZROAZRDFQEWCICMBBLDBGF4ROSSFIHFAG7FQ54NT3CAMQUQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.372589289 (520h40m26.243672324s) [noise=true , address=145.79.227.37:4565 , piece_id=5X3VB4R3PIYJ3UCJFHSIWDI54QIYKXWDHKZDK54F6IYOBFJNTJOA, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.372965185 (520h40m26.24404823s) [noise=true , address=145.79.227.38:4978 , piece_id=LZCQ3FVBSGQGE3AJD2QW7TPS7UPIQYMHJNQW63QG7NRPON4O3JFA, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.373828608 (520h40m26.244911653s) [noise=true , address=45.91.116.13:4155 , piece_id=6ER3WHBVRZE5ALSWIHXNOQ4CSYOZFBUWA6GVDGJW2GYGWMMIK6UQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.374572645 (520h40m26.245655691s) [noise=true , address=145.79.227.45:4230 , piece_id=OXCWUTWNYQ334UIJD5VUPN3TZQN3SQ7YKBQ67ZXEYEMREK7H2N6Q, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.375050425 (520h40m26.24613346s) [noise=true , address=145.79.227.36:4518 , piece_id=NZJZPMVGEHJW5BHOBOESZDRSHFLB5KVJ3T7WKLE3BFQTH76HDGMQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.376116815 (520h40m26.247199861s) [noise=true , address=45.91.116.8:4048 , piece_id=YDJTJTZNPTQRV7LICVSWUUIMV4KYWNTVJL636U2XZYOPHG4FCUMA, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.377967610 (520h40m26.249050645s) [noise=true , address=45.91.116.14:4191 , piece_id=LHWP6YCBTRSE5N54D26COMCFCFQVAUXJLFZSCWLT7TJ7FBU4FEOQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.380198808 (520h40m26.251281843s) [noise=true , address=145.79.227.17:4971 , piece_id=3MSQ6NEWTBVD5SXGYS22DP2VZCSCYPO3YBUKOKBTRAGCXQSOEBNQ, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.381752877 (520h40m26.252835912s) [noise=true , address=77.90.31.17:4770 , piece_id=6F7GSMMQMYA4MLO4BDS4B3FMTPHWONA32CQRZKXFGXW6GKKSUM5A, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.382993770 (520h40m26.254076815s) [noise=true , address=145.79.227.48:4301 , piece_id=ANROBDYIVE5HE5M7SCLNWSUVALYCPT3QXKIBRBB3NHNTTLZGHE6A, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:26.383103348 (520h40m26.254186393s) [noise=true , address=145.79.227.13:4802 , piece_id=PBANAYWFBVU2WPHTON4X3NDEYNFVVO4JHTSADJCMCVRKUMKHP6VA, seg_pos={0 0}, upload=456775]: Uploading piece... Dec 27 08:09:28.091833944 (520h40m27.962916979s) [upload=456775]: Next returned nil segment Dec 27 08:09:28.091865424 (520h40m27.962948459s) [upload=456775]: Waiting for error group managing segments... Dec 27 08:09:28.379562512 (520h40m28.250645548s) [noise=true , address=145.79.227.28:4726 , piece_id=VVDYZ7UUS2YLVCMWII7CST5ZQRVNYC5ADO36TVQHULXXSE4ZJUJQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.384439298 (520h40m28.255522343s) [noise=true , address=145.79.227.36:4518 , piece_id=NZJZPMVGEHJW5BHOBOESZDRSHFLB5KVJ3T7WKLE3BFQTH76HDGMQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.392125994 (520h40m28.263209029s) [noise=true , address=145.79.227.51:4409 , piece_id=SIHKY6TAUP6TWJ3XHQDPCIFE5ZNJQK7DS7NOEBQ33ELPMXL2RECQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.394945653 (520h40m28.266028698s) [noise=true , address=145.79.227.15:4873 , piece_id=KMFBA4MYMF7LLBJ3REURD5VLSX6RD23WU4MHLX2KUNLPRI235ASQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.397607591 (520h40m28.268690636s) [noise=true , address=145.79.227.13:4802 , piece_id=PBANAYWFBVU2WPHTON4X3NDEYNFVVO4JHTSADJCMCVRKUMKHP6VA, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.402472534 (520h40m28.273555579s) [noise=true , address=145.79.227.50:4374 , piece_id=B5NAGZROAZRDFQEWCICMBBLDBGF4ROSSFIHFAG7FQ54NT3CAMQUQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.406075776 (520h40m28.277158811s) [noise=true , address=145.79.227.37:4565 , piece_id=5X3VB4R3PIYJ3UCJFHSIWDI54QIYKXWDHKZDK54F6IYOBFJNTJOA, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.406451221 (520h40m28.277534266s) [noise=true , address=145.79.227.45:4230 , piece_id=OXCWUTWNYQ334UIJD5VUPN3TZQN3SQ7YKBQ67ZXEYEMREK7H2N6Q, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.407955666 (520h40m28.279038701s) [noise=true , address=145.79.227.17:4971 , piece_id=3MSQ6NEWTBVD5SXGYS22DP2VZCSCYPO3YBUKOKBTRAGCXQSOEBNQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.410335908 (520h40m28.281418943s) [noise=true , address=145.79.227.38:4978 , piece_id=LZCQ3FVBSGQGE3AJD2QW7TPS7UPIQYMHJNQW63QG7NRPON4O3JFA, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.412509025 (520h40m28.28359207s) [noise=true , address=145.79.227.48:4301 , piece_id=ANROBDYIVE5HE5M7SCLNWSUVALYCPT3QXKIBRBB3NHNTTLZGHE6A, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.610500316 (520h40m28.481583361s) [noise=true , address=45.91.116.14:4191 , piece_id=LHWP6YCBTRSE5N54D26COMCFCFQVAUXJLFZSCWLT7TJ7FBU4FEOQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.636259874 (520h40m28.507342909s) [noise=true , address=45.91.116.8:4048 , piece_id=YDJTJTZNPTQRV7LICVSWUUIMV4KYWNTVJL636U2XZYOPHG4FCUMA, seg_pos={0 0}, upload=456775]: Done uploading piece. err: <nil> Dec 27 08:09:28.636282227 (520h40m28.507365262s) [seg_pos={0 0}, upload=456775]: Segment reached optimal threshold of 13 pieces. Dec 27 08:09:28.636505191 (520h40m28.507588226s) [noise=true , address=77.90.31.17:4770 , piece_id=6F7GSMMQMYA4MLO4BDS4B3FMTPHWONA32CQRZKXFGXW6GKKSUM5A, seg_pos={0 0}, upload=456775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 08:09:28.636511935 (520h40m28.50759497s) [noise=true , address=77.90.31.17:4770 , piece_id=6F7GSMMQMYA4MLO4BDS4B3FMTPHWONA32CQRZKXFGXW6GKKSUM5A, seg_pos={0 0}, upload=456775]: Long tail cancellation due to optimal threshold Dec 27 08:09:28.636637673 (520h40m28.507720718s) [noise=true , address=77.90.31.14:4663 , piece_id=QZKGYYSTSPQAVRKQHTW4G6CNYBZXSDVSADUKOKXLUN5EGWRMGKSA, seg_pos={0 0}, upload=456775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 08:09:28.636658383 (520h40m28.507741418s) [noise=true , address=77.90.31.14:4663 , piece_id=QZKGYYSTSPQAVRKQHTW4G6CNYBZXSDVSADUKOKXLUN5EGWRMGKSA, seg_pos={0 0}, upload=456775]: Long tail cancellation due to optimal threshold Dec 27 08:09:28.636694602 (520h40m28.507777637s) [noise=true , address=45.91.116.13:4155 , piece_id=6ER3WHBVRZE5ALSWIHXNOQ4CSYOZFBUWA6GVDGJW2GYGWMMIK6UQ, seg_pos={0 0}, upload=456775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 08:09:28.636703439 (520h40m28.507786474s) [noise=true , address=45.91.116.13:4155 , piece_id=6ER3WHBVRZE5ALSWIHXNOQ4CSYOZFBUWA6GVDGJW2GYGWMMIK6UQ, seg_pos={0 0}, upload=456775]: Long tail cancellation due to optimal threshold Dec 27 08:09:28.636721744 (520h40m28.507804779s) [seg_pos={0 0}, upload=456775]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:09:28.636835851 (520h40m28.507918886s) [upload=456775]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:09:28.636841942 (520h40m28.507924978s) [upload=456775]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:09:28.928333505 (520h40m28.79941654s) [upload=456775]: Done uploading segments... Dec 27 08:09:28.928353103 (520h40m28.799436148s) [upload=456775]: Upload finished. err: <nil> Duration: 30.003362484s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 43.478713568s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:09:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 43.478741541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 39.128231973s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:08:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 39.128246531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........X..X..........X......X.....X.X....X.X......X......X.] duration: 5.720721595s trace: https://jaeger.dev.storj.io/trace/2ca34f9b24b69c42 FAIL (2025-12-27 08:08:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: ZJGEEQJ4SB3WYEG6EDQAC4WTOK5ICLR5V422EP5G7GIKN7Z5Q4GQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 53S45LPQBFSUKHTTBL3QK5L3P7APSQXFXZ5JAN4L6FRIJMCYT5BA) context deadline exceeded ======LOGS====== gments... Dec 27 08:07:37.206023429 (520h38m37.077106464s) [upload=456748]: Got next segment. Inline: false Dec 27 08:07:37.206069747 (520h38m37.077152842s) [upload=456748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:07:37.206076229 (520h38m37.077159265s) [upload=456748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:07:37.452939743 (520h38m37.324022778s) [seg_pos={0 0}, upload=456748]: Begin upload segment... Dec 27 08:07:37.452989738 (520h38m37.324072773s) [seg_pos={0 0}, upload=456748]: Done begin upload segment. Dec 27 08:07:37.453029123 (520h38m37.324112158s) [noise=true , address=145.79.227.42:4122 , piece_id=XETUQ7S6JQ2XVSSLCQQYRZBYU3D7P4GZJMQH4SYKVTNGT6FIPALQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.453046466 (520h38m37.324129501s) [noise=true , address=145.79.227.46:4601 , piece_id=37RRIH7VNVSD6UCNTUAL4DQC25Y5FGPNVDZBCWQRIQBXRYV5XODA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.453590001 (520h38m37.324673036s) [noise=true , address=45.91.116.11:4118 , piece_id=53S45LPQBFSUKHTTBL3QK5L3P7APSQXFXZ5JAN4L6FRIJMCYT5BA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.453986556 (520h38m37.325069591s) [noise=true , address=45.91.116.14:4191 , piece_id=UJBZFN5ZFBNHZCIZPX5W7MFZC4LJCFJRGKM5XLFNCP6TU7QRDITA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.454359867 (520h38m37.325442902s) [noise=true , address=145.79.227.5:4222 , piece_id=NMYCVUG74FIL2NER3ZLYHN2RQDSHSAMY5JMS43VJDG25IPDW2RPQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.454765600 (520h38m37.325848635s) [noise=true , address=145.79.227.17:4971 , piece_id=NG3IQCSIMLHPWWQYSWQ4FVLQVA2H4ZLFBL53X5KKHF3XXACA3IVQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.455123261 (520h38m37.326206296s) [noise=true , address=45.91.116.4:4940 , piece_id=HILIJEMK6LL5OUPP6XHUH3NQGJLAH5VNGGF276IZV2ASLLXEY4QA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.455559992 (520h38m37.326643027s) [noise=true , address=145.79.227.4:4185 , piece_id=3NSGA532HPVVVORZJLZSZLCXW6ON3WODZFZXTAIG3QBOZCIIOQJA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.455989590 (520h38m37.327072625s) [noise=true , address=145.79.227.37:4565 , piece_id=CV7IE4IZWTEC74SWCZ3GTMI3C6JLFBIXCMFI27TV2GWIF2SDFXFA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.456828257 (520h38m37.327911292s) [noise=true , address=145.79.227.50:4374 , piece_id=XKPXBQVRPU4BWUWEUHNQDIHVPYM7GU4QTH7HDFAD5QUH3WNG4GUA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.458091803 (520h38m37.329174848s) [noise=true , address=45.91.116.8:4047 , piece_id=L5K2NAOS4QCQS62ZVNVS7K5WNSPREP6LNGVWFICCKW5DB6F7PJXQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.459878896 (520h38m37.330961941s) [noise=true , address=77.90.31.19:4841 , piece_id=ZUKYCH2FPKBEOQ2X2E3JAV5ON6TYAPS4P2V2WXQPXEJ2DKU5C6WA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.461247372 (520h38m37.332330417s) [noise=true , address=45.91.116.7:4009 , piece_id=ZJGEEQJ4SB3WYEG6EDQAC4WTOK5ICLR5V422EP5G7GIKN7Z5Q4GQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.462581973 (520h38m37.333665008s) [noise=true , address=145.79.227.35:4481 , piece_id=WD7WC4WNHUXSJ7XRWKPBEUN3E3C5YTDQ6RAIILUCLPER2OGDNAZQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.465361626 (520h38m37.336444671s) [noise=true , address=145.79.227.51:4411 , piece_id=6GASU65BPEVYUARI665DQCOJTJOS65KK5TTBSK2SHEG5776W62FA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.467851968 (520h38m37.338935003s) [noise=true , address=145.79.227.11:4730 , piece_id=4IKHGEAG6OI3OBZNYIA64HFSVEZPB4GTSIUKHGKH2ATYSSIV4LXQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:39.196817514 (520h38m39.067900549s) [upload=456748]: Next returned nil segment Dec 27 08:07:39.196853052 (520h38m39.067936097s) [upload=456748]: Waiting for error group managing segments... Dec 27 08:07:39.490631759 (520h38m39.361714824s) [noise=true , address=145.79.227.4:4185 , piece_id=3NSGA532HPVVVORZJLZSZLCXW6ON3WODZFZXTAIG3QBOZCIIOQJA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.505280772 (520h38m39.376363807s) [noise=true , address=145.79.227.35:4481 , piece_id=WD7WC4WNHUXSJ7XRWKPBEUN3E3C5YTDQ6RAIILUCLPER2OGDNAZQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.506188010 (520h38m39.377271045s) [noise=true , address=145.79.227.11:4730 , piece_id=4IKHGEAG6OI3OBZNYIA64HFSVEZPB4GTSIUKHGKH2ATYSSIV4LXQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.516023137 (520h38m39.387106172s) [noise=true , address=145.79.227.5:4222 , piece_id=NMYCVUG74FIL2NER3ZLYHN2RQDSHSAMY5JMS43VJDG25IPDW2RPQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.531588676 (520h38m39.402671711s) [noise=true , address=145.79.227.46:4601 , piece_id=37RRIH7VNVSD6UCNTUAL4DQC25Y5FGPNVDZBCWQRIQBXRYV5XODA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.567653018 (520h38m39.438736053s) [noise=true , address=145.79.227.17:4971 , piece_id=NG3IQCSIMLHPWWQYSWQ4FVLQVA2H4ZLFBL53X5KKHF3XXACA3IVQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.581520303 (520h38m39.452603338s) [noise=true , address=145.79.227.37:4565 , piece_id=CV7IE4IZWTEC74SWCZ3GTMI3C6JLFBIXCMFI27TV2GWIF2SDFXFA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.582176463 (520h38m39.453259508s) [noise=true , address=145.79.227.42:4122 , piece_id=XETUQ7S6JQ2XVSSLCQQYRZBYU3D7P4GZJMQH4SYKVTNGT6FIPALQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.586998625 (520h38m39.45808166s) [noise=true , address=145.79.227.51:4411 , piece_id=6GASU65BPEVYUARI665DQCOJTJOS65KK5TTBSK2SHEG5776W62FA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.603551524 (520h38m39.474634559s) [noise=true , address=145.79.227.50:4374 , piece_id=XKPXBQVRPU4BWUWEUHNQDIHVPYM7GU4QTH7HDFAD5QUH3WNG4GUA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.813055569 (520h38m39.684138614s) [noise=true , address=45.91.116.11:4118 , piece_id=53S45LPQBFSUKHTTBL3QK5L3P7APSQXFXZ5JAN4L6FRIJMCYT5BA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.834977722 (520h38m39.706060757s) [noise=true , address=45.91.116.7:4009 , piece_id=ZJGEEQJ4SB3WYEG6EDQAC4WTOK5ICLR5V422EP5G7GIKN7Z5Q4GQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.837882934 (520h38m39.708965999s) [noise=true , address=45.91.116.14:4191 , piece_id=UJBZFN5ZFBNHZCIZPX5W7MFZC4LJCFJRGKM5XLFNCP6TU7QRDITA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.837902191 (520h38m39.708985236s) [seg_pos={0 0}, upload=456748]: Segment reached optimal threshold of 13 pieces. Dec 27 08:07:39.838160111 (520h38m39.709243167s) [noise=true , address=77.90.31.19:4841 , piece_id=ZUKYCH2FPKBEOQ2X2E3JAV5ON6TYAPS4P2V2WXQPXEJ2DKU5C6WA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 27 08:07:39.838172475 (520h38m39.70925555s) [noise=true , address=77.90.31.19:4841 , piece_id=ZUKYCH2FPKBEOQ2X2E3JAV5ON6TYAPS4P2V2WXQPXEJ2DKU5C6WA, seg_pos={0 0}, upload=456748]: Long tail cancellation due to optimal threshold Dec 27 08:07:39.838460744 (520h38m39.709543799s) [noise=true , address=45.91.116.8:4047 , piece_id=L5K2NAOS4QCQS62ZVNVS7K5WNSPREP6LNGVWFICCKW5DB6F7PJXQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:38328->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:38328->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38328->45.91.116.8:4047: use of closed network connection Dec 27 08:07:39.838469931 (520h38m39.709552977s) [noise=true , address=45.91.116.8:4047 , piece_id=L5K2NAOS4QCQS62ZVNVS7K5WNSPREP6LNGVWFICCKW5DB6F7PJXQ, seg_pos={0 0}, upload=456748]: Long tail cancellation due to optimal threshold Dec 27 08:07:39.838578308 (520h38m39.709661353s) [noise=true , address=45.91.116.4:4940 , piece_id=HILIJEMK6LL5OUPP6XHUH3NQGJLAH5VNGGF276IZV2ASLLXEY4QA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:46500->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:46500->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46500->45.91.116.4:4940: use of closed network connection Dec 27 08:07:39.838587826 (520h38m39.709670871s) [noise=true , address=45.91.116.4:4940 , piece_id=HILIJEMK6LL5OUPP6XHUH3NQGJLAH5VNGGF276IZV2ASLLXEY4QA, seg_pos={0 0}, upload=456748]: Long tail cancellation due to optimal threshold Dec 27 08:07:39.838604318 (520h38m39.709687353s) [seg_pos={0 0}, upload=456748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:07:39.838679571 (520h38m39.709762616s) [upload=456748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:07:39.838685432 (520h38m39.709768467s) [upload=456748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:07:40.130101961 (520h38m40.001185006s) [upload=456748]: Done uploading segments... Dec 27 08:07:40.130119874 (520h38m40.001202909s) [upload=456748]: Upload finished. err: <nil> Duration: 5.720745721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........X..X..........X......X.....X.X....X.X......X......X] duration: 30.003461261s trace: https://jaeger.dev.storj.io/trace/2ca34f9b24b69c42 FAIL (2025-12-27 08:08:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: ZJGEEQJ4SB3WYEG6EDQAC4WTOK5ICLR5V422EP5G7GIKN7Z5Q4GQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 53S45LPQBFSUKHTTBL3QK5L3P7APSQXFXZ5JAN4L6FRIJMCYT5BA) context deadline exceeded ======LOGS====== gments... Dec 27 08:07:37.206023429 (520h38m37.077106464s) [upload=456748]: Got next segment. Inline: false Dec 27 08:07:37.206069747 (520h38m37.077152842s) [upload=456748]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:07:37.206076229 (520h38m37.077159265s) [upload=456748]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:07:37.452939743 (520h38m37.324022778s) [seg_pos={0 0}, upload=456748]: Begin upload segment... Dec 27 08:07:37.452989738 (520h38m37.324072773s) [seg_pos={0 0}, upload=456748]: Done begin upload segment. Dec 27 08:07:37.453029123 (520h38m37.324112158s) [noise=true , address=145.79.227.42:4122 , piece_id=XETUQ7S6JQ2XVSSLCQQYRZBYU3D7P4GZJMQH4SYKVTNGT6FIPALQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.453046466 (520h38m37.324129501s) [noise=true , address=145.79.227.46:4601 , piece_id=37RRIH7VNVSD6UCNTUAL4DQC25Y5FGPNVDZBCWQRIQBXRYV5XODA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.453590001 (520h38m37.324673036s) [noise=true , address=45.91.116.11:4118 , piece_id=53S45LPQBFSUKHTTBL3QK5L3P7APSQXFXZ5JAN4L6FRIJMCYT5BA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.453986556 (520h38m37.325069591s) [noise=true , address=45.91.116.14:4191 , piece_id=UJBZFN5ZFBNHZCIZPX5W7MFZC4LJCFJRGKM5XLFNCP6TU7QRDITA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.454359867 (520h38m37.325442902s) [noise=true , address=145.79.227.5:4222 , piece_id=NMYCVUG74FIL2NER3ZLYHN2RQDSHSAMY5JMS43VJDG25IPDW2RPQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.454765600 (520h38m37.325848635s) [noise=true , address=145.79.227.17:4971 , piece_id=NG3IQCSIMLHPWWQYSWQ4FVLQVA2H4ZLFBL53X5KKHF3XXACA3IVQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.455123261 (520h38m37.326206296s) [noise=true , address=45.91.116.4:4940 , piece_id=HILIJEMK6LL5OUPP6XHUH3NQGJLAH5VNGGF276IZV2ASLLXEY4QA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.455559992 (520h38m37.326643027s) [noise=true , address=145.79.227.4:4185 , piece_id=3NSGA532HPVVVORZJLZSZLCXW6ON3WODZFZXTAIG3QBOZCIIOQJA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.455989590 (520h38m37.327072625s) [noise=true , address=145.79.227.37:4565 , piece_id=CV7IE4IZWTEC74SWCZ3GTMI3C6JLFBIXCMFI27TV2GWIF2SDFXFA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.456828257 (520h38m37.327911292s) [noise=true , address=145.79.227.50:4374 , piece_id=XKPXBQVRPU4BWUWEUHNQDIHVPYM7GU4QTH7HDFAD5QUH3WNG4GUA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.458091803 (520h38m37.329174848s) [noise=true , address=45.91.116.8:4047 , piece_id=L5K2NAOS4QCQS62ZVNVS7K5WNSPREP6LNGVWFICCKW5DB6F7PJXQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.459878896 (520h38m37.330961941s) [noise=true , address=77.90.31.19:4841 , piece_id=ZUKYCH2FPKBEOQ2X2E3JAV5ON6TYAPS4P2V2WXQPXEJ2DKU5C6WA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.461247372 (520h38m37.332330417s) [noise=true , address=45.91.116.7:4009 , piece_id=ZJGEEQJ4SB3WYEG6EDQAC4WTOK5ICLR5V422EP5G7GIKN7Z5Q4GQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.462581973 (520h38m37.333665008s) [noise=true , address=145.79.227.35:4481 , piece_id=WD7WC4WNHUXSJ7XRWKPBEUN3E3C5YTDQ6RAIILUCLPER2OGDNAZQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.465361626 (520h38m37.336444671s) [noise=true , address=145.79.227.51:4411 , piece_id=6GASU65BPEVYUARI665DQCOJTJOS65KK5TTBSK2SHEG5776W62FA, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:37.467851968 (520h38m37.338935003s) [noise=true , address=145.79.227.11:4730 , piece_id=4IKHGEAG6OI3OBZNYIA64HFSVEZPB4GTSIUKHGKH2ATYSSIV4LXQ, seg_pos={0 0}, upload=456748]: Uploading piece... Dec 27 08:07:39.196817514 (520h38m39.067900549s) [upload=456748]: Next returned nil segment Dec 27 08:07:39.196853052 (520h38m39.067936097s) [upload=456748]: Waiting for error group managing segments... Dec 27 08:07:39.490631759 (520h38m39.361714824s) [noise=true , address=145.79.227.4:4185 , piece_id=3NSGA532HPVVVORZJLZSZLCXW6ON3WODZFZXTAIG3QBOZCIIOQJA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.505280772 (520h38m39.376363807s) [noise=true , address=145.79.227.35:4481 , piece_id=WD7WC4WNHUXSJ7XRWKPBEUN3E3C5YTDQ6RAIILUCLPER2OGDNAZQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.506188010 (520h38m39.377271045s) [noise=true , address=145.79.227.11:4730 , piece_id=4IKHGEAG6OI3OBZNYIA64HFSVEZPB4GTSIUKHGKH2ATYSSIV4LXQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.516023137 (520h38m39.387106172s) [noise=true , address=145.79.227.5:4222 , piece_id=NMYCVUG74FIL2NER3ZLYHN2RQDSHSAMY5JMS43VJDG25IPDW2RPQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.531588676 (520h38m39.402671711s) [noise=true , address=145.79.227.46:4601 , piece_id=37RRIH7VNVSD6UCNTUAL4DQC25Y5FGPNVDZBCWQRIQBXRYV5XODA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.567653018 (520h38m39.438736053s) [noise=true , address=145.79.227.17:4971 , piece_id=NG3IQCSIMLHPWWQYSWQ4FVLQVA2H4ZLFBL53X5KKHF3XXACA3IVQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.581520303 (520h38m39.452603338s) [noise=true , address=145.79.227.37:4565 , piece_id=CV7IE4IZWTEC74SWCZ3GTMI3C6JLFBIXCMFI27TV2GWIF2SDFXFA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.582176463 (520h38m39.453259508s) [noise=true , address=145.79.227.42:4122 , piece_id=XETUQ7S6JQ2XVSSLCQQYRZBYU3D7P4GZJMQH4SYKVTNGT6FIPALQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.586998625 (520h38m39.45808166s) [noise=true , address=145.79.227.51:4411 , piece_id=6GASU65BPEVYUARI665DQCOJTJOS65KK5TTBSK2SHEG5776W62FA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.603551524 (520h38m39.474634559s) [noise=true , address=145.79.227.50:4374 , piece_id=XKPXBQVRPU4BWUWEUHNQDIHVPYM7GU4QTH7HDFAD5QUH3WNG4GUA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.813055569 (520h38m39.684138614s) [noise=true , address=45.91.116.11:4118 , piece_id=53S45LPQBFSUKHTTBL3QK5L3P7APSQXFXZ5JAN4L6FRIJMCYT5BA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.834977722 (520h38m39.706060757s) [noise=true , address=45.91.116.7:4009 , piece_id=ZJGEEQJ4SB3WYEG6EDQAC4WTOK5ICLR5V422EP5G7GIKN7Z5Q4GQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.837882934 (520h38m39.708965999s) [noise=true , address=45.91.116.14:4191 , piece_id=UJBZFN5ZFBNHZCIZPX5W7MFZC4LJCFJRGKM5XLFNCP6TU7QRDITA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: <nil> Dec 27 08:07:39.837902191 (520h38m39.708985236s) [seg_pos={0 0}, upload=456748]: Segment reached optimal threshold of 13 pieces. Dec 27 08:07:39.838160111 (520h38m39.709243167s) [noise=true , address=77.90.31.19:4841 , piece_id=ZUKYCH2FPKBEOQ2X2E3JAV5ON6TYAPS4P2V2WXQPXEJ2DKU5C6WA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 27 08:07:39.838172475 (520h38m39.70925555s) [noise=true , address=77.90.31.19:4841 , piece_id=ZUKYCH2FPKBEOQ2X2E3JAV5ON6TYAPS4P2V2WXQPXEJ2DKU5C6WA, seg_pos={0 0}, upload=456748]: Long tail cancellation due to optimal threshold Dec 27 08:07:39.838460744 (520h38m39.709543799s) [noise=true , address=45.91.116.8:4047 , piece_id=L5K2NAOS4QCQS62ZVNVS7K5WNSPREP6LNGVWFICCKW5DB6F7PJXQ, seg_pos={0 0}, upload=456748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:38328->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:38328->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38328->45.91.116.8:4047: use of closed network connection Dec 27 08:07:39.838469931 (520h38m39.709552977s) [noise=true , address=45.91.116.8:4047 , piece_id=L5K2NAOS4QCQS62ZVNVS7K5WNSPREP6LNGVWFICCKW5DB6F7PJXQ, seg_pos={0 0}, upload=456748]: Long tail cancellation due to optimal threshold Dec 27 08:07:39.838578308 (520h38m39.709661353s) [noise=true , address=45.91.116.4:4940 , piece_id=HILIJEMK6LL5OUPP6XHUH3NQGJLAH5VNGGF276IZV2ASLLXEY4QA, seg_pos={0 0}, upload=456748]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:46500->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:46500->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46500->45.91.116.4:4940: use of closed network connection Dec 27 08:07:39.838587826 (520h38m39.709670871s) [noise=true , address=45.91.116.4:4940 , piece_id=HILIJEMK6LL5OUPP6XHUH3NQGJLAH5VNGGF276IZV2ASLLXEY4QA, seg_pos={0 0}, upload=456748]: Long tail cancellation due to optimal threshold Dec 27 08:07:39.838604318 (520h38m39.709687353s) [seg_pos={0 0}, upload=456748]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:07:39.838679571 (520h38m39.709762616s) [upload=456748]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:07:39.838685432 (520h38m39.709768467s) [upload=456748]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:07:40.130101961 (520h38m40.001185006s) [upload=456748]: Done uploading segments... Dec 27 08:07:40.130119874 (520h38m40.001202909s) [upload=456748]: Upload finished. err: <nil> Duration: 30.003494374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 44.596181926s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:07:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 44.59620535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X..X..........X......X.....X.X....X.X......X......] duration: 6.876134628s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:07:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 6.876165385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X..X..........X......X.....X.X....X.X......X.....] duration: 5.919807535s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:06:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 5.919835157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 51.010098543s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:06:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 51.010138179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X...........X..X..........X......X.....X.X....X.X......X....] duration: 5.870777304s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 5.870804235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X...........X..X..........X......X.....X.X....X.X......X...] duration: 6.161071016s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:05:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 6.161101685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 52.175922974s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:05:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 52.175939886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X...........X..X..........X......X.....X.X....X.X......X..] duration: 5.733471224s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:05:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 5.733529054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 32.802525254s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:04:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 32.802550812s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.434383424s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:04:38 UTC) orbiter warning: context deadline exceeded Duration: 3.434408102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X...........X..X..........X......X.....X.X....X.X......X.] duration: 6.193204149s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 6.193265165s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.471373627s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:04:09 UTC) orbiter warning: context deadline exceeded Duration: 3.471404928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 41.195811751s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:03:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 41.195826711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..X...........X..X..........X......X.....X.X....X.X......X] duration: 30.003456367s trace: https://jaeger.dev.storj.io/trace/55d1437c3bb9f5be FAIL (2025-12-27 08:03:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A) context deadline exceeded ======LOGS====== Dec 27 08:03:11.463528331 (520h34m11.334611376s) [upload=456688]: Starting upload Dec 27 08:03:11.463626117 (520h34m11.334709162s) [upload=456688]: Done starting upload Dec 27 08:03:11.463678226 (520h34m11.334761271s) [upload=456688]: Uploading segments... Dec 27 08:03:11.463890560 (520h34m11.334973605s) [upload=456688]: Got next segment. Inline: false Dec 27 08:03:11.463951326 (520h34m11.335034371s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 08:03:11.463957808 (520h34m11.335040853s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 08:03:11.712320416 (520h34m11.583403451s) [seg_pos={0 0}, upload=456688]: Begin upload segment... Dec 27 08:03:11.712380791 (520h34m11.583463836s) [seg_pos={0 0}, upload=456688]: Done begin upload segment. Dec 27 08:03:11.712425356 (520h34m11.583508401s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713165937 (520h34m11.584248982s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.713866811 (520h34m11.584949996s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714171541 (520h34m11.585254587s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714733381 (520h34m11.585816426s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.714761053 (520h34m11.585844098s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715168920 (520h34m11.586251965s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715448703 (520h34m11.586531748s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.715763202 (520h34m11.586846247s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716124350 (520h34m11.587207395s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716208680 (520h34m11.587291725s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716668886 (520h34m11.587751921s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.716675899 (520h34m11.587758944s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717083636 (520h34m11.588166671s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.717477777 (520h34m11.588560822s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:11.718833538 (520h34m11.589916583s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Uploading piece... Dec 27 08:03:13.553326052 (520h34m13.424409087s) [upload=456688]: Next returned nil segment Dec 27 08:03:13.553341813 (520h34m13.424424848s) [upload=456688]: Waiting for error group managing segments... Dec 27 08:03:13.870690469 (520h34m13.741773504s) [noise=true , address=145.79.227.11:4730 , piece_id=Q6XE67BODUS6KHKC5SXHPXMPKDHSDX4YL3RTB6ZWQ5R2VLIHK62Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.876173309 (520h34m13.747256354s) [noise=true , address=145.79.227.52:4445 , piece_id=NKUVHFJ3Q2N7U2AONH7M5MERI2ZEYBGZCWOHB4ZCDUBMRG6XAQIQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.877079204 (520h34m13.748162239s) [noise=true , address=145.79.227.17:4971 , piece_id=UVY4TH4AV7ACI2I2XTXFLAV5CWLMQLVOWZGFCAPFWK7X5IVAZQPQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879125981 (520h34m13.750209027s) [noise=true , address=145.79.227.51:4410 , piece_id=OQJJH7C7SMON5FDOGWFIVTYMMFL267I6BE7FZNFQ7SEW3UI62QYQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.879652654 (520h34m13.750735699s) [noise=true , address=145.79.227.21:4474 , piece_id=WJMYB3GIII5ZMR2UERVGQGXZGFPWUSOKMC5LAQQBTDGOBSBHSKTA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.886478151 (520h34m13.757561196s) [noise=true , address=145.79.227.41:4085 , piece_id=RC4HMXBSTKP2FMNF2CJDP7T5KEBW3OMRIMML5SW2ADY32VWTEOJQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.888107994 (520h34m13.759191029s) [noise=true , address=145.79.227.37:4566 , piece_id=4KLX4JCVF6QNIOZE2KNWOGEQDZ4IZEXVZA55KFDHEX72JR5LECVA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.891764637 (520h34m13.762847682s) [noise=true , address=145.79.227.28:4725 , piece_id=GLPBCTRNSKOGPGD7SD2GT42PGCXC4NXPSRVTEPKYTWEJIS6JNZOQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.905868593 (520h34m13.776951638s) [noise=true , address=145.79.227.27:4689 , piece_id=EMPFIQ64WGQUAAGJTXHBRFZHFVNBCIRXRZ3YALAA5NCDXTQLAN6Q, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.926822812 (520h34m13.797905857s) [noise=true , address=77.90.31.16:4736 , piece_id=4CYENULSJA6H7CNCIMSFTA2WC2KPFGXLIPN6SS7G3OFQH743IHGQ, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.964453807 (520h34m13.835536842s) [noise=true , address=45.91.116.13:4154 , piece_id=HO4SG3PSYTBGVXSHETLF7CAKJ3HAUUYR6NJBBJLE3CIRDHDJFI7A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:13.998006236 (520h34m13.869089281s) [noise=true , address=45.91.116.14:4189 , piece_id=ZEM6AS666BAHZDQLY2B5VTOFPQOTDEGV33Q34A6G7G5465UOQW3A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006499107 (520h34m13.877582142s) [noise=true , address=77.90.31.14:4664 , piece_id=THBBP6RNJVPK7XTA24L4P54UBKCTLMZDXJGL4FAZQJP5WH4BC7RA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: <nil> Dec 27 08:03:14.006581044 (520h34m13.877664079s) [seg_pos={0 0}, upload=456688]: Segment reached optimal threshold of 13 pieces. Dec 27 08:03:14.006950557 (520h34m13.878033592s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.006963632 (520h34m13.878046677s) [noise=true , address=45.91.116.7:4012 , piece_id=UA524BWQUBVK2F6IBOHVDRH2RG6ETOYL2VSYHPMVDZVTHETPVJ2A, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007160687 (520h34m13.878243732s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007170906 (520h34m13.878253951s) [noise=true , address=145.79.227.38:4978 , piece_id=EWZFOVG3JCW3X5NWCJ4RGES74MNQE4QJX74FTDYBSXX3PY4P3JIA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007469996 (520h34m13.878553041s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4): protocol: expected piece hash; context canceled; EOF Dec 27 08:03:14.007486838 (520h34m13.878569873s) [noise=true , address=145.79.227.10:4403 , piece_id=G24MHBA7B7IKGEFI7RBUHOLSUX5KV6WPL3OWL73GLOFNOMTNGOQA, seg_pos={0 0}, upload=456688]: Long tail cancellation due to optimal threshold Dec 27 08:03:14.007554357 (520h34m13.878637392s) [seg_pos={0 0}, upload=456688]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 08:03:14.007653846 (520h34m13.878736891s) [upload=456688]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 08:03:14.007659577 (520h34m13.878742622s) [upload=456688]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 08:03:14.301648624 (520h34m14.172731669s) [upload=456688]: Done uploading segments... Dec 27 08:03:14.301718717 (520h34m14.172801762s) [upload=456688]: Upload finished. err: <nil> Duration: 30.003482566s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.511543868s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:03:17 UTC) orbiter warning: context deadline exceeded Duration: 3.511566703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 52.157161739s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 52.157175215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.599136167s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:02:42 UTC) orbiter warning: context deadline exceeded Duration: 3.599223373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X...........X..X..........X......X.....X.X....X.X......] duration: 5.651274117s trace: https://jaeger.dev.storj.io/trace/287a4ebef2251592 FAIL (2025-12-27 08:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=456625]: Uploading segments... Dec 27 07:58:57.979002063 (520h29m57.850085098s) [upload=456625]: Got next segment. Inline: false Dec 27 07:58:57.979038682 (520h29m57.850121727s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:58:57.979044032 (520h29m57.850127067s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:58:58.227505018 (520h29m58.098588074s) [seg_pos={0 0}, upload=456625]: Begin upload segment... Dec 27 07:58:58.227570422 (520h29m58.098653467s) [seg_pos={0 0}, upload=456625]: Done begin upload segment. Dec 27 07:58:58.227626980 (520h29m58.098710055s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.227682186 (520h29m58.098765231s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228318577 (520h29m58.099401612s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228826044 (520h29m58.099909079s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229258537 (520h29m58.100341572s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229649361 (520h29m58.100732396s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.231098210 (520h29m58.102181245s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.232565965 (520h29m58.10364901s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.233973094 (520h29m58.105056139s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.236428480 (520h29m58.107511525s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.238049236 (520h29m58.109132271s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239394858 (520h29m58.110477893s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239704688 (520h29m58.110787733s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241499817 (520h29m58.112582852s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241870362 (520h29m58.112953397s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.243204853 (520h29m58.114287898s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:59.945769246 (520h29m59.816852281s) [upload=456625]: Next returned nil segment Dec 27 07:58:59.945789574 (520h29m59.816872609s) [upload=456625]: Waiting for error group managing segments... Dec 27 07:59:00.232413327 (520h30m0.103496372s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.234033813 (520h30m0.105116858s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240162332 (520h30m0.111245377s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240360048 (520h30m0.111443113s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.245446433 (520h30m0.116529479s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.247852174 (520h30m0.118935209s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.252683905 (520h30m0.12376696s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.272717732 (520h30m0.143800777s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.275389781 (520h30m0.146472816s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.439609858 (520h30m0.310692894s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.532294743 (520h30m0.403377778s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.570394481 (520h30m0.441477526s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265560385 (520h30m1.13664343s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265592346 (520h30m1.136675421s) [seg_pos={0 0}, upload=456625]: Segment reached optimal threshold of 13 pieces. Dec 27 07:59:01.265920190 (520h30m1.137003235s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection Dec 27 07:59:01.265932152 (520h30m1.137015197s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.265986446 (520h30m1.137069491s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection Dec 27 07:59:01.266017465 (520h30m1.13710051s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266244477 (520h30m1.137327522s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 07:59:01.266267772 (520h30m1.137350837s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266309391 (520h30m1.137392436s) [seg_pos={0 0}, upload=456625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:59:01.266422055 (520h30m1.13750512s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:59:01.266429189 (520h30m1.137512234s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:59:01.572785245 (520h30m1.44386829s) [upload=456625]: Done uploading segments... Dec 27 07:59:01.572807848 (520h30m1.443890893s) [upload=456625]: Upload finished. err: <nil> Duration: 5.651301498s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.385901868s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:02:19 UTC) orbiter warning: context deadline exceeded Duration: 3.385926884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 42.405720413s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:02:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 42.40574079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..X...........X..X..........X......X.....X.X....X.X.....] duration: 5.875661243s trace: https://jaeger.dev.storj.io/trace/287a4ebef2251592 FAIL (2025-12-27 08:01:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=456625]: Uploading segments... Dec 27 07:58:57.979002063 (520h29m57.850085098s) [upload=456625]: Got next segment. Inline: false Dec 27 07:58:57.979038682 (520h29m57.850121727s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:58:57.979044032 (520h29m57.850127067s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:58:58.227505018 (520h29m58.098588074s) [seg_pos={0 0}, upload=456625]: Begin upload segment... Dec 27 07:58:58.227570422 (520h29m58.098653467s) [seg_pos={0 0}, upload=456625]: Done begin upload segment. Dec 27 07:58:58.227626980 (520h29m58.098710055s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.227682186 (520h29m58.098765231s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228318577 (520h29m58.099401612s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228826044 (520h29m58.099909079s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229258537 (520h29m58.100341572s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229649361 (520h29m58.100732396s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.231098210 (520h29m58.102181245s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.232565965 (520h29m58.10364901s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.233973094 (520h29m58.105056139s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.236428480 (520h29m58.107511525s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.238049236 (520h29m58.109132271s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239394858 (520h29m58.110477893s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239704688 (520h29m58.110787733s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241499817 (520h29m58.112582852s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241870362 (520h29m58.112953397s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.243204853 (520h29m58.114287898s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:59.945769246 (520h29m59.816852281s) [upload=456625]: Next returned nil segment Dec 27 07:58:59.945789574 (520h29m59.816872609s) [upload=456625]: Waiting for error group managing segments... Dec 27 07:59:00.232413327 (520h30m0.103496372s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.234033813 (520h30m0.105116858s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240162332 (520h30m0.111245377s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240360048 (520h30m0.111443113s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.245446433 (520h30m0.116529479s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.247852174 (520h30m0.118935209s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.252683905 (520h30m0.12376696s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.272717732 (520h30m0.143800777s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.275389781 (520h30m0.146472816s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.439609858 (520h30m0.310692894s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.532294743 (520h30m0.403377778s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.570394481 (520h30m0.441477526s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265560385 (520h30m1.13664343s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265592346 (520h30m1.136675421s) [seg_pos={0 0}, upload=456625]: Segment reached optimal threshold of 13 pieces. Dec 27 07:59:01.265920190 (520h30m1.137003235s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection Dec 27 07:59:01.265932152 (520h30m1.137015197s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.265986446 (520h30m1.137069491s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection Dec 27 07:59:01.266017465 (520h30m1.13710051s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266244477 (520h30m1.137327522s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 07:59:01.266267772 (520h30m1.137350837s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266309391 (520h30m1.137392436s) [seg_pos={0 0}, upload=456625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:59:01.266422055 (520h30m1.13750512s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:59:01.266429189 (520h30m1.137512234s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:59:01.572785245 (520h30m1.44386829s) [upload=456625]: Done uploading segments... Dec 27 07:59:01.572807848 (520h30m1.443890893s) [upload=456625]: Upload finished. err: <nil> Duration: 5.875688535s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.550732691s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:01:54 UTC) orbiter warning: context deadline exceeded Duration: 3.550954072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..X...........X..X..........X......X.....X.X....X.X....] duration: 5.908770216s trace: https://jaeger.dev.storj.io/trace/287a4ebef2251592 FAIL (2025-12-27 08:01:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=456625]: Uploading segments... Dec 27 07:58:57.979002063 (520h29m57.850085098s) [upload=456625]: Got next segment. Inline: false Dec 27 07:58:57.979038682 (520h29m57.850121727s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:58:57.979044032 (520h29m57.850127067s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:58:58.227505018 (520h29m58.098588074s) [seg_pos={0 0}, upload=456625]: Begin upload segment... Dec 27 07:58:58.227570422 (520h29m58.098653467s) [seg_pos={0 0}, upload=456625]: Done begin upload segment. Dec 27 07:58:58.227626980 (520h29m58.098710055s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.227682186 (520h29m58.098765231s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228318577 (520h29m58.099401612s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228826044 (520h29m58.099909079s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229258537 (520h29m58.100341572s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229649361 (520h29m58.100732396s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.231098210 (520h29m58.102181245s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.232565965 (520h29m58.10364901s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.233973094 (520h29m58.105056139s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.236428480 (520h29m58.107511525s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.238049236 (520h29m58.109132271s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239394858 (520h29m58.110477893s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239704688 (520h29m58.110787733s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241499817 (520h29m58.112582852s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241870362 (520h29m58.112953397s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.243204853 (520h29m58.114287898s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:59.945769246 (520h29m59.816852281s) [upload=456625]: Next returned nil segment Dec 27 07:58:59.945789574 (520h29m59.816872609s) [upload=456625]: Waiting for error group managing segments... Dec 27 07:59:00.232413327 (520h30m0.103496372s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.234033813 (520h30m0.105116858s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240162332 (520h30m0.111245377s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240360048 (520h30m0.111443113s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.245446433 (520h30m0.116529479s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.247852174 (520h30m0.118935209s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.252683905 (520h30m0.12376696s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.272717732 (520h30m0.143800777s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.275389781 (520h30m0.146472816s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.439609858 (520h30m0.310692894s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.532294743 (520h30m0.403377778s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.570394481 (520h30m0.441477526s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265560385 (520h30m1.13664343s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265592346 (520h30m1.136675421s) [seg_pos={0 0}, upload=456625]: Segment reached optimal threshold of 13 pieces. Dec 27 07:59:01.265920190 (520h30m1.137003235s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection Dec 27 07:59:01.265932152 (520h30m1.137015197s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.265986446 (520h30m1.137069491s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection Dec 27 07:59:01.266017465 (520h30m1.13710051s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266244477 (520h30m1.137327522s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 07:59:01.266267772 (520h30m1.137350837s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266309391 (520h30m1.137392436s) [seg_pos={0 0}, upload=456625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:59:01.266422055 (520h30m1.13750512s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:59:01.266429189 (520h30m1.137512234s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:59:01.572785245 (520h30m1.44386829s) [upload=456625]: Done uploading segments... Dec 27 07:59:01.572807848 (520h30m1.443890893s) [upload=456625]: Upload finished. err: <nil> Duration: 5.908785465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 47.381449864s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 47.38147423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X....X..X...........X..X..........X......X.....X.X....X.X...] duration: 20.085626742s trace: https://jaeger.dev.storj.io/trace/287a4ebef2251592 FAIL (2025-12-27 08:01:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=456625]: Uploading segments... Dec 27 07:58:57.979002063 (520h29m57.850085098s) [upload=456625]: Got next segment. Inline: false Dec 27 07:58:57.979038682 (520h29m57.850121727s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:58:57.979044032 (520h29m57.850127067s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:58:58.227505018 (520h29m58.098588074s) [seg_pos={0 0}, upload=456625]: Begin upload segment... Dec 27 07:58:58.227570422 (520h29m58.098653467s) [seg_pos={0 0}, upload=456625]: Done begin upload segment. Dec 27 07:58:58.227626980 (520h29m58.098710055s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.227682186 (520h29m58.098765231s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228318577 (520h29m58.099401612s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228826044 (520h29m58.099909079s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229258537 (520h29m58.100341572s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229649361 (520h29m58.100732396s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.231098210 (520h29m58.102181245s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.232565965 (520h29m58.10364901s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.233973094 (520h29m58.105056139s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.236428480 (520h29m58.107511525s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.238049236 (520h29m58.109132271s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239394858 (520h29m58.110477893s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239704688 (520h29m58.110787733s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241499817 (520h29m58.112582852s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241870362 (520h29m58.112953397s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.243204853 (520h29m58.114287898s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:59.945769246 (520h29m59.816852281s) [upload=456625]: Next returned nil segment Dec 27 07:58:59.945789574 (520h29m59.816872609s) [upload=456625]: Waiting for error group managing segments... Dec 27 07:59:00.232413327 (520h30m0.103496372s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.234033813 (520h30m0.105116858s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240162332 (520h30m0.111245377s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240360048 (520h30m0.111443113s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.245446433 (520h30m0.116529479s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.247852174 (520h30m0.118935209s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.252683905 (520h30m0.12376696s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.272717732 (520h30m0.143800777s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.275389781 (520h30m0.146472816s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.439609858 (520h30m0.310692894s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.532294743 (520h30m0.403377778s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.570394481 (520h30m0.441477526s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265560385 (520h30m1.13664343s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265592346 (520h30m1.136675421s) [seg_pos={0 0}, upload=456625]: Segment reached optimal threshold of 13 pieces. Dec 27 07:59:01.265920190 (520h30m1.137003235s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection Dec 27 07:59:01.265932152 (520h30m1.137015197s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.265986446 (520h30m1.137069491s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection Dec 27 07:59:01.266017465 (520h30m1.13710051s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266244477 (520h30m1.137327522s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 07:59:01.266267772 (520h30m1.137350837s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266309391 (520h30m1.137392436s) [seg_pos={0 0}, upload=456625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:59:01.266422055 (520h30m1.13750512s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:59:01.266429189 (520h30m1.137512234s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:59:01.572785245 (520h30m1.44386829s) [upload=456625]: Done uploading segments... Dec 27 07:59:01.572807848 (520h30m1.443890893s) [upload=456625]: Upload finished. err: <nil> Duration: 20.085658162s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 4.803882222s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:01:27 UTC) orbiter warning: context deadline exceeded Duration: 4.80390767s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.406374723s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 08:00:40 UTC) orbiter warning: context deadline exceeded Duration: 3.406403388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X....X..X...........X..X..........X......X.....X.X....X.X..] duration: 6.345097384s trace: https://jaeger.dev.storj.io/trace/287a4ebef2251592 FAIL (2025-12-27 08:00:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== load=456625]: Uploading segments... Dec 27 07:58:57.979002063 (520h29m57.850085098s) [upload=456625]: Got next segment. Inline: false Dec 27 07:58:57.979038682 (520h29m57.850121727s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:58:57.979044032 (520h29m57.850127067s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:58:58.227505018 (520h29m58.098588074s) [seg_pos={0 0}, upload=456625]: Begin upload segment... Dec 27 07:58:58.227570422 (520h29m58.098653467s) [seg_pos={0 0}, upload=456625]: Done begin upload segment. Dec 27 07:58:58.227626980 (520h29m58.098710055s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.227682186 (520h29m58.098765231s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228318577 (520h29m58.099401612s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.228826044 (520h29m58.099909079s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229258537 (520h29m58.100341572s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.229649361 (520h29m58.100732396s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.231098210 (520h29m58.102181245s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.232565965 (520h29m58.10364901s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.233973094 (520h29m58.105056139s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.236428480 (520h29m58.107511525s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.238049236 (520h29m58.109132271s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239394858 (520h29m58.110477893s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.239704688 (520h29m58.110787733s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241499817 (520h29m58.112582852s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.241870362 (520h29m58.112953397s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:58.243204853 (520h29m58.114287898s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Uploading piece... Dec 27 07:58:59.945769246 (520h29m59.816852281s) [upload=456625]: Next returned nil segment Dec 27 07:58:59.945789574 (520h29m59.816872609s) [upload=456625]: Waiting for error group managing segments... Dec 27 07:59:00.232413327 (520h30m0.103496372s) [noise=true , address=145.79.227.37:4567 , piece_id=M64HBSEXMTQ4LWFLMVRL2G3JEGVGNVKTPPUKJFSUVX7KNACRWYCQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.234033813 (520h30m0.105116858s) [noise=true , address=145.79.227.27:4689 , piece_id=PU6JCPB5TK3ZZQ2RIG2UUYJS2IARDAMRCM76LTEIDCKYFUJ6QEUQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240162332 (520h30m0.111245377s) [noise=true , address=145.79.227.21:4474 , piece_id=WF6BSOFTWKR6GKUW6WCNXULCKMXN3E2UKCKVFHT2X3ZF44WOSQAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.240360048 (520h30m0.111443113s) [noise=true , address=145.79.227.43:4158 , piece_id=L3ISPPRHYPRWOY5GYI2B7NT56UGQ7Q7ADQAC3SGLNYE3XJ7UG6VA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.245446433 (520h30m0.116529479s) [noise=true , address=145.79.227.8:4330 , piece_id=F26WIJPBIKO7AEA75X6I3GNMHIZBKFK53NJFUS4NMNC35SSAURNA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.247852174 (520h30m0.118935209s) [noise=true , address=145.79.227.50:4375 , piece_id=ETMPXEGBZ77D4BNZUZJFVGXGC6NGF2PV7LL23Z4SBJXFHFTZEKGA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.252683905 (520h30m0.12376696s) [noise=true , address=145.79.227.15:4874 , piece_id=FJQQKO4YHGL6T75HFTMNSKM6OHZIVXAT5HCZEG5LGT73WSKRLJAQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.272717732 (520h30m0.143800777s) [noise=true , address=145.79.227.35:4481 , piece_id=QYUIIQ5GGZ3LIO5F6TDCXE5FSNWZTOTLJ7WFA6C5DFFEUX6UCAEA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.275389781 (520h30m0.146472816s) [noise=true , address=145.79.227.29:4942 , piece_id=K3GEJS5CR62LB5LA3H6GLZWCJTBHI734CDV5WVO4KTEJVJDCS5ZA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.439609858 (520h30m0.310692894s) [noise=true , address=77.90.31.12:4590 , piece_id=4EPE4MTVFUMPBFGFQZAUNR375ZFRL3WMTTQ3WWDXEPGNI6667CRA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.532294743 (520h30m0.403377778s) [noise=true , address=45.91.116.8:4045 , piece_id=5NO6IRNX7WHYCHSDGP5TJIF52JR2ESEAWHTAQFCPNHMFCTOOPTHQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:00.570394481 (520h30m0.441477526s) [noise=true , address=77.90.31.21:4880 , piece_id=2XBP2OXLHXEDQNB35EOOQBVWNLPJQKEBQBSSAEY5M5BDENU2EUQQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265560385 (520h30m1.13664343s) [noise=true , address=45.91.116.9:4264 , piece_id=ADAVRIX2VBXJXSLJME4OENDMBAMGQYRERZP6LAZZT3BHNCHO56VQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: <nil> Dec 27 07:59:01.265592346 (520h30m1.136675421s) [seg_pos={0 0}, upload=456625]: Segment reached optimal threshold of 13 pieces. Dec 27 07:59:01.265920190 (520h30m1.137003235s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49142->45.91.116.6:4974: use of closed network connection Dec 27 07:59:01.265932152 (520h30m1.137015197s) [noise=true , address=45.91.116.6:4974 , piece_id=XU4QIAG4KRPAHA24IUV6JKCBGFWYQSQO5VVDLJQHGQQ55IINJNPQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.265986446 (520h30m1.137069491s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55000->45.91.116.14:4190: use of closed network connection Dec 27 07:59:01.266017465 (520h30m1.13710051s) [noise=true , address=45.91.116.14:4190 , piece_id=7QBYCH7A3C2JUEVULNPWDQPUOU6UF22RJZSOHOEWWSA2VN4Q7IHA, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266244477 (520h30m1.137327522s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 07:59:01.266267772 (520h30m1.137350837s) [noise=true , address=45.91.116.4:4937 , piece_id=DVCVERXQ73X7BQC3LNCOKICOXB6AIKRVGKMSQESGEG7XXUHCL5ZQ, seg_pos={0 0}, upload=456625]: Long tail cancellation due to optimal threshold Dec 27 07:59:01.266309391 (520h30m1.137392436s) [seg_pos={0 0}, upload=456625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:59:01.266422055 (520h30m1.13750512s) [upload=456625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:59:01.266429189 (520h30m1.137512234s) [upload=456625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:59:01.572785245 (520h30m1.44386829s) [upload=456625]: Done uploading segments... Dec 27 07:59:01.572807848 (520h30m1.443890893s) [upload=456625]: Upload finished. err: <nil> Duration: 6.34513189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 41.716766352s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 08:00:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 41.716785118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 34.041809341s trace: https://jaeger.dev.storj.io/trace/748bb2e8032f1b13 FAIL (2025-12-27 07:59:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243299587 (519h48m46.114382622s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.243776155 (519h48m46.11485919s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244139927 (519h48m46.115222972s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244210681 (519h48m46.115293727s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244408789 (519h48m46.115491824s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244704642 (519h48m46.115787677s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244738436 (519h48m46.115821481s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.244977872 (519h48m46.116060907s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245250962 (519h48m46.116333998s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245276181 (519h48m46.116359226s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245699927 (519h48m46.116782973s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.245752597 (519h48m46.116835642s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246130878 (519h48m46.117213923s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246289841 (519h48m46.117372876s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.246766519 (519h48m46.117849564s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:46.247271019 (519h48m46.118354064s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Uploading piece... Dec 27 07:17:47.528719771 (519h48m47.399802806s) [upload=456028]: Next returned nil segment Dec 27 07:17:47.528735712 (519h48m47.399818747s) [upload=456028]: Waiting for error group managing segments... Dec 27 07:17:47.865953654 (519h48m47.737036699s) [noise=true , address=64.49.13.140:4507 , piece_id=VYPOBEBC3SNTAP3BNZC4MKZGLGFGG6PXBJVUKR6L6HEOKPN67VJA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.113976823 (519h48m47.985059858s) [noise=true , address=145.79.227.43:4184 , piece_id=M62QMJUOC3FW54VWKXP3PLH6OKGQ3E3DTIANUHBD56QVLQYZBMEQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.149431214 (519h48m48.02051426s) [noise=true , address=145.79.227.34:4934 , piece_id=BYSUU6DIDB4EOPW4MDK5MMRMTHKDYTHDK2JLX224Q57FBACN7RSQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.166685268 (519h48m48.037768314s) [noise=true , address=145.79.227.16:4926 , piece_id=6PU4RDCO2GQP5XQ5PFF44LWFGHIYOTXJT5STXQCY343SY4VQEKHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.365735135 (519h48m48.23681817s) [noise=true , address=77.90.31.18:4823 , piece_id=6AW5NOHEWYH3VC6FJLE7VEYUVLJT5VCGRY2ZGBJ3NGAHO43WXQBQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.382770603 (519h48m48.253853638s) [noise=true , address=77.90.31.19:4866 , piece_id=P7ZFGPMDVY5TFVGBFLM7GYNVOTXRJIKO5GCYTDMLY33VUYMKLFWA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.397079610 (519h48m48.268162645s) [noise=true , address=77.90.31.14:4677 , piece_id=54FA6GKXJ3673ODFUBMAJWFDZ23IOTDRKYOTJRXIA5L7JQLX7OVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.443322074 (519h48m48.314405119s) [noise=true , address=45.91.116.5:4329 , piece_id=FCSKBNDIXWXN5AO5QBCCPAFEFDXSQFJAI3FADMCYJUY5BZOOEYHA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:48.714036701 (519h48m48.585119737s) [noise=true , address=77.90.31.15:4716 , piece_id=TZO4JO3PXQV2AR7QBYMFETJTN3ABGXZUUFCIN7YUFUHB7352CROA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.052790548 (519h48m48.923873583s) [noise=true , address=145.79.227.30:4782 , piece_id=JU5STXD2NUVISB5NFLRUNZZNRURY7LXYQNAY6YMJS75FLWKOIDJQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.077483107 (519h48m48.948566152s) [noise=true , address=145.79.227.40:4081 , piece_id=E6TUQHSZK7PFRNMXNJTC4JZFLF2NNAI5CDKSGKLRSJTLU7CDZASQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.110300857 (519h48m48.981383903s) [noise=true , address=145.79.227.44:4209 , piece_id=HQ72WEPXO7DTVAOPLNUUGBACD4SJU4ONF2DTYH3F2E35MUYVYGLQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.248016752 (519h48m49.119099787s) [noise=true , address=77.90.31.11:4567 , piece_id=HJPSXDOYCCRXO3LOAK7D2OVTO6SEHOHGSLNRCVI3OJSG6YFLRBAA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462976834 (519h48m49.334059869s) [noise=true , address=77.90.31.13:4643 , piece_id=POSXD7Q4HSVLD5D5TCY3KCE3TZ7GD2FZXX6CNMSJVJFBGWXEKERA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: <nil> Dec 27 07:17:49.462998304 (519h48m49.334081349s) [seg_pos={0 0}, upload=456028]: Segment reached optimal threshold of 14 pieces. Dec 27 07:17:49.463204086 (519h48m49.334287121s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463210249 (519h48m49.334293294s) [noise=true , address=193.149.164.143:4701 , piece_id=B5NUF5MN26V24AMDWKWMPUVQ53SAZY335ATKS7OBHNZRAKPOF57Q, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463411371 (519h48m49.334494406s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463429857 (519h48m49.334512892s) [noise=true , address=45.91.116.7:4027 , piece_id=2MWXSACXZHUQKPVVL6AXOGHBQPAGBYFZXHMK6X5TURAEH2ZXYHVQ, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463442791 (519h48m49.334525826s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155fvDXZdpQeHetitv17WCVVrP1bGswdoqJECQLSe66kXRhBPo): protocol: read tcp 188.166.225.170:43022->77.90.31.17:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:17:49.463449103 (519h48m49.334532158s) [noise=true , address=77.90.31.17:4783 , piece_id=NO6TSOB23D66PGFW73P5Z72Y4F2I36OOHRL44NTUUHSELD7MIFQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463519086 (519h48m49.334602131s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA): protocol: expected piece hash; context canceled; EOF Dec 27 07:17:49.463526281 (519h48m49.334609316s) [noise=true , address=45.91.116.11:4144 , piece_id=IZ75WFGRGOGAD2R7U3PC7LIRGJZPS6XKGAO7MO2OZV7BNMJQVCQA, seg_pos={0 0}, upload=456028]: Long tail cancellation due to optimal threshold Dec 27 07:17:49.463587737 (519h48m49.334670772s) [seg_pos={0 0}, upload=456028]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:17:49.463714519 (519h48m49.334797554s) [upload=456028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:17:49.463718937 (519h48m49.334801972s) [upload=456028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:17:49.763381046 (519h48m49.634464081s) [upload=456028]: Done uploading segments... Dec 27 07:17:49.763401095 (519h48m49.63448413s) [upload=456028]: Upload finished. err: <nil> Duration: 34.04182423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.502677766s trace: https://jaeger.dev.storj.io/trace/25f7d5ea91653a56 FAIL (2025-12-27 07:59:35 UTC) orbiter warning: context deadline exceeded Duration: 3.502691541s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)