Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XX........X....X.X.....] duration: 6.23082285s trace: https://jaeger.dev.storj.io/trace/5dab66cd6a1c3e1f FAIL (2025-12-30 21:22:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ) context deadline exceeded ======LOGS====== Dec 30 21:18:15.038535656 (605h49m14.909618691s) [upload=530966]: Starting upload Dec 30 21:18:15.038593327 (605h49m14.909676362s) [upload=530966]: Done starting upload Dec 30 21:18:15.038772347 (605h49m14.909855382s) [upload=530966]: Uploading segments... Dec 30 21:18:15.038808045 (605h49m14.90989108s) [upload=530966]: Got next segment. Inline: false Dec 30 21:18:15.038836740 (605h49m14.909919785s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:18:15.038841619 (605h49m14.909924664s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:18:15.288031620 (605h49m15.159114665s) [seg_pos={0 0}, upload=530966]: Begin upload segment... Dec 30 21:18:15.288104969 (605h49m15.159188014s) [seg_pos={0 0}, upload=530966]: Done begin upload segment. Dec 30 21:18:15.288164493 (605h49m15.159247528s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288175323 (605h49m15.159258358s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288372088 (605h49m15.159455123s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288771518 (605h49m15.159854553s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.289068294 (605h49m15.160151329s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.290630729 (605h49m15.161713794s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291035529 (605h49m15.162118584s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291378322 (605h49m15.162461357s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.292676684 (605h49m15.163759729s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.294120654 (605h49m15.165203689s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.295450887 (605h49m15.166533922s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.297065291 (605h49m15.168148326s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298458484 (605h49m15.169541529s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298531132 (605h49m15.169614177s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.299892464 (605h49m15.170975529s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.300767750 (605h49m15.171850796s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:16.997604903 (605h49m16.868687948s) [upload=530966]: Next returned nil segment Dec 30 21:18:16.997621605 (605h49m16.86870465s) [upload=530966]: Waiting for error group managing segments... Dec 30 21:18:17.275743075 (605h49m17.14682611s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.280241811 (605h49m17.151324846s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305053826 (605h49m17.176136861s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305171220 (605h49m17.176254265s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.342799479 (605h49m17.213882514s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.369239945 (605h49m17.24032299s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.370450030 (605h49m17.241533075s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.546745219 (605h49m17.417828264s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.577326469 (605h49m17.448409504s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.579343340 (605h49m17.450426375s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.598317289 (605h49m17.469400345s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.610488385 (605h49m17.48157143s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677438386 (605h49m17.548521421s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677464406 (605h49m17.548547451s) [seg_pos={0 0}, upload=530966]: Segment reached optimal threshold of 13 pieces. Dec 30 21:18:17.677709983 (605h49m17.548793028s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.677718699 (605h49m17.548801745s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678074226 (605h49m17.549157271s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678083164 (605h49m17.549166209s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678238670 (605h49m17.549321715s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678248548 (605h49m17.549331593s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678269308 (605h49m17.549352343s) [seg_pos={0 0}, upload=530966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:18:17.678356864 (605h49m17.549439899s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:18:17.678364389 (605h49m17.549447424s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:18:17.965275506 (605h49m17.836358541s) [upload=530966]: Done uploading segments... Dec 30 21:18:17.965295543 (605h49m17.836378578s) [upload=530966]: Upload finished. err: <nil> Duration: 6.230850502s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X......X...............................................] duration: 3.772714215s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:21:51 UTC) orbiter warning: context deadline exceeded Duration: 3.772740214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................XX........X....X.X....] duration: 6.448723809s trace: https://jaeger.dev.storj.io/trace/5dab66cd6a1c3e1f FAIL (2025-12-30 21:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ) context deadline exceeded ======LOGS====== Dec 30 21:18:15.038535656 (605h49m14.909618691s) [upload=530966]: Starting upload Dec 30 21:18:15.038593327 (605h49m14.909676362s) [upload=530966]: Done starting upload Dec 30 21:18:15.038772347 (605h49m14.909855382s) [upload=530966]: Uploading segments... Dec 30 21:18:15.038808045 (605h49m14.90989108s) [upload=530966]: Got next segment. Inline: false Dec 30 21:18:15.038836740 (605h49m14.909919785s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:18:15.038841619 (605h49m14.909924664s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:18:15.288031620 (605h49m15.159114665s) [seg_pos={0 0}, upload=530966]: Begin upload segment... Dec 30 21:18:15.288104969 (605h49m15.159188014s) [seg_pos={0 0}, upload=530966]: Done begin upload segment. Dec 30 21:18:15.288164493 (605h49m15.159247528s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288175323 (605h49m15.159258358s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288372088 (605h49m15.159455123s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288771518 (605h49m15.159854553s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.289068294 (605h49m15.160151329s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.290630729 (605h49m15.161713794s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291035529 (605h49m15.162118584s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291378322 (605h49m15.162461357s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.292676684 (605h49m15.163759729s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.294120654 (605h49m15.165203689s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.295450887 (605h49m15.166533922s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.297065291 (605h49m15.168148326s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298458484 (605h49m15.169541529s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298531132 (605h49m15.169614177s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.299892464 (605h49m15.170975529s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.300767750 (605h49m15.171850796s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:16.997604903 (605h49m16.868687948s) [upload=530966]: Next returned nil segment Dec 30 21:18:16.997621605 (605h49m16.86870465s) [upload=530966]: Waiting for error group managing segments... Dec 30 21:18:17.275743075 (605h49m17.14682611s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.280241811 (605h49m17.151324846s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305053826 (605h49m17.176136861s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305171220 (605h49m17.176254265s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.342799479 (605h49m17.213882514s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.369239945 (605h49m17.24032299s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.370450030 (605h49m17.241533075s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.546745219 (605h49m17.417828264s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.577326469 (605h49m17.448409504s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.579343340 (605h49m17.450426375s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.598317289 (605h49m17.469400345s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.610488385 (605h49m17.48157143s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677438386 (605h49m17.548521421s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677464406 (605h49m17.548547451s) [seg_pos={0 0}, upload=530966]: Segment reached optimal threshold of 13 pieces. Dec 30 21:18:17.677709983 (605h49m17.548793028s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.677718699 (605h49m17.548801745s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678074226 (605h49m17.549157271s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678083164 (605h49m17.549166209s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678238670 (605h49m17.549321715s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678248548 (605h49m17.549331593s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678269308 (605h49m17.549352343s) [seg_pos={0 0}, upload=530966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:18:17.678356864 (605h49m17.549439899s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:18:17.678364389 (605h49m17.549447424s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:18:17.965275506 (605h49m17.836358541s) [upload=530966]: Done uploading segments... Dec 30 21:18:17.965295543 (605h49m17.836378578s) [upload=530966]: Upload finished. err: <nil> Duration: 6.448758495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......X......X..............................................] duration: 3.513639065s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:21:22 UTC) orbiter warning: context deadline exceeded Duration: 3.513669612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......X......X.............................................] duration: 3.849467409s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:21:04 UTC) orbiter warning: context deadline exceeded Duration: 3.849495963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XX........X....X.X...] duration: 6.592264817s trace: https://jaeger.dev.storj.io/trace/5dab66cd6a1c3e1f FAIL (2025-12-30 21:20:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ) context deadline exceeded ======LOGS====== Dec 30 21:18:15.038535656 (605h49m14.909618691s) [upload=530966]: Starting upload Dec 30 21:18:15.038593327 (605h49m14.909676362s) [upload=530966]: Done starting upload Dec 30 21:18:15.038772347 (605h49m14.909855382s) [upload=530966]: Uploading segments... Dec 30 21:18:15.038808045 (605h49m14.90989108s) [upload=530966]: Got next segment. Inline: false Dec 30 21:18:15.038836740 (605h49m14.909919785s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:18:15.038841619 (605h49m14.909924664s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:18:15.288031620 (605h49m15.159114665s) [seg_pos={0 0}, upload=530966]: Begin upload segment... Dec 30 21:18:15.288104969 (605h49m15.159188014s) [seg_pos={0 0}, upload=530966]: Done begin upload segment. Dec 30 21:18:15.288164493 (605h49m15.159247528s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288175323 (605h49m15.159258358s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288372088 (605h49m15.159455123s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288771518 (605h49m15.159854553s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.289068294 (605h49m15.160151329s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.290630729 (605h49m15.161713794s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291035529 (605h49m15.162118584s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291378322 (605h49m15.162461357s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.292676684 (605h49m15.163759729s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.294120654 (605h49m15.165203689s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.295450887 (605h49m15.166533922s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.297065291 (605h49m15.168148326s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298458484 (605h49m15.169541529s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298531132 (605h49m15.169614177s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.299892464 (605h49m15.170975529s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.300767750 (605h49m15.171850796s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:16.997604903 (605h49m16.868687948s) [upload=530966]: Next returned nil segment Dec 30 21:18:16.997621605 (605h49m16.86870465s) [upload=530966]: Waiting for error group managing segments... Dec 30 21:18:17.275743075 (605h49m17.14682611s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.280241811 (605h49m17.151324846s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305053826 (605h49m17.176136861s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305171220 (605h49m17.176254265s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.342799479 (605h49m17.213882514s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.369239945 (605h49m17.24032299s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.370450030 (605h49m17.241533075s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.546745219 (605h49m17.417828264s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.577326469 (605h49m17.448409504s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.579343340 (605h49m17.450426375s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.598317289 (605h49m17.469400345s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.610488385 (605h49m17.48157143s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677438386 (605h49m17.548521421s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677464406 (605h49m17.548547451s) [seg_pos={0 0}, upload=530966]: Segment reached optimal threshold of 13 pieces. Dec 30 21:18:17.677709983 (605h49m17.548793028s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.677718699 (605h49m17.548801745s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678074226 (605h49m17.549157271s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678083164 (605h49m17.549166209s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678238670 (605h49m17.549321715s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678248548 (605h49m17.549331593s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678269308 (605h49m17.549352343s) [seg_pos={0 0}, upload=530966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:18:17.678356864 (605h49m17.549439899s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:18:17.678364389 (605h49m17.549447424s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:18:17.965275506 (605h49m17.836358541s) [upload=530966]: Done uploading segments... Dec 30 21:18:17.965295543 (605h49m17.836378578s) [upload=530966]: Upload finished. err: <nil> Duration: 6.592300606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........X......X............................................] duration: 3.433210177s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:20:41 UTC) orbiter warning: context deadline exceeded Duration: 3.43323763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......................................XX........X....X.X..] duration: 5.773537029s trace: https://jaeger.dev.storj.io/trace/5dab66cd6a1c3e1f FAIL (2025-12-30 21:20:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ) context deadline exceeded ======LOGS====== Dec 30 21:18:15.038535656 (605h49m14.909618691s) [upload=530966]: Starting upload Dec 30 21:18:15.038593327 (605h49m14.909676362s) [upload=530966]: Done starting upload Dec 30 21:18:15.038772347 (605h49m14.909855382s) [upload=530966]: Uploading segments... Dec 30 21:18:15.038808045 (605h49m14.90989108s) [upload=530966]: Got next segment. Inline: false Dec 30 21:18:15.038836740 (605h49m14.909919785s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:18:15.038841619 (605h49m14.909924664s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:18:15.288031620 (605h49m15.159114665s) [seg_pos={0 0}, upload=530966]: Begin upload segment... Dec 30 21:18:15.288104969 (605h49m15.159188014s) [seg_pos={0 0}, upload=530966]: Done begin upload segment. Dec 30 21:18:15.288164493 (605h49m15.159247528s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288175323 (605h49m15.159258358s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288372088 (605h49m15.159455123s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288771518 (605h49m15.159854553s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.289068294 (605h49m15.160151329s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.290630729 (605h49m15.161713794s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291035529 (605h49m15.162118584s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291378322 (605h49m15.162461357s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.292676684 (605h49m15.163759729s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.294120654 (605h49m15.165203689s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.295450887 (605h49m15.166533922s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.297065291 (605h49m15.168148326s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298458484 (605h49m15.169541529s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298531132 (605h49m15.169614177s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.299892464 (605h49m15.170975529s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.300767750 (605h49m15.171850796s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:16.997604903 (605h49m16.868687948s) [upload=530966]: Next returned nil segment Dec 30 21:18:16.997621605 (605h49m16.86870465s) [upload=530966]: Waiting for error group managing segments... Dec 30 21:18:17.275743075 (605h49m17.14682611s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.280241811 (605h49m17.151324846s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305053826 (605h49m17.176136861s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305171220 (605h49m17.176254265s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.342799479 (605h49m17.213882514s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.369239945 (605h49m17.24032299s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.370450030 (605h49m17.241533075s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.546745219 (605h49m17.417828264s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.577326469 (605h49m17.448409504s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.579343340 (605h49m17.450426375s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.598317289 (605h49m17.469400345s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.610488385 (605h49m17.48157143s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677438386 (605h49m17.548521421s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677464406 (605h49m17.548547451s) [seg_pos={0 0}, upload=530966]: Segment reached optimal threshold of 13 pieces. Dec 30 21:18:17.677709983 (605h49m17.548793028s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.677718699 (605h49m17.548801745s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678074226 (605h49m17.549157271s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678083164 (605h49m17.549166209s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678238670 (605h49m17.549321715s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678248548 (605h49m17.549331593s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678269308 (605h49m17.549352343s) [seg_pos={0 0}, upload=530966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:18:17.678356864 (605h49m17.549439899s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:18:17.678364389 (605h49m17.549447424s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:18:17.965275506 (605h49m17.836358541s) [upload=530966]: Done uploading segments... Dec 30 21:18:17.965295543 (605h49m17.836378578s) [upload=530966]: Upload finished. err: <nil> Duration: 5.773567616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........X......X...........................................] duration: 3.672044297s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:19:52 UTC) orbiter warning: context deadline exceeded Duration: 3.672068543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X......X..........................................] duration: 3.70302496s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:19:02 UTC) orbiter warning: context deadline exceeded Duration: 3.703051431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......................................XX........X....X.X.] duration: 6.097347385s trace: https://jaeger.dev.storj.io/trace/5dab66cd6a1c3e1f FAIL (2025-12-30 21:18:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ) context deadline exceeded ======LOGS====== Dec 30 21:18:15.038535656 (605h49m14.909618691s) [upload=530966]: Starting upload Dec 30 21:18:15.038593327 (605h49m14.909676362s) [upload=530966]: Done starting upload Dec 30 21:18:15.038772347 (605h49m14.909855382s) [upload=530966]: Uploading segments... Dec 30 21:18:15.038808045 (605h49m14.90989108s) [upload=530966]: Got next segment. Inline: false Dec 30 21:18:15.038836740 (605h49m14.909919785s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:18:15.038841619 (605h49m14.909924664s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:18:15.288031620 (605h49m15.159114665s) [seg_pos={0 0}, upload=530966]: Begin upload segment... Dec 30 21:18:15.288104969 (605h49m15.159188014s) [seg_pos={0 0}, upload=530966]: Done begin upload segment. Dec 30 21:18:15.288164493 (605h49m15.159247528s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288175323 (605h49m15.159258358s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288372088 (605h49m15.159455123s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288771518 (605h49m15.159854553s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.289068294 (605h49m15.160151329s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.290630729 (605h49m15.161713794s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291035529 (605h49m15.162118584s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291378322 (605h49m15.162461357s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.292676684 (605h49m15.163759729s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.294120654 (605h49m15.165203689s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.295450887 (605h49m15.166533922s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.297065291 (605h49m15.168148326s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298458484 (605h49m15.169541529s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298531132 (605h49m15.169614177s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.299892464 (605h49m15.170975529s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.300767750 (605h49m15.171850796s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:16.997604903 (605h49m16.868687948s) [upload=530966]: Next returned nil segment Dec 30 21:18:16.997621605 (605h49m16.86870465s) [upload=530966]: Waiting for error group managing segments... Dec 30 21:18:17.275743075 (605h49m17.14682611s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.280241811 (605h49m17.151324846s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305053826 (605h49m17.176136861s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305171220 (605h49m17.176254265s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.342799479 (605h49m17.213882514s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.369239945 (605h49m17.24032299s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.370450030 (605h49m17.241533075s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.546745219 (605h49m17.417828264s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.577326469 (605h49m17.448409504s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.579343340 (605h49m17.450426375s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.598317289 (605h49m17.469400345s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.610488385 (605h49m17.48157143s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677438386 (605h49m17.548521421s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677464406 (605h49m17.548547451s) [seg_pos={0 0}, upload=530966]: Segment reached optimal threshold of 13 pieces. Dec 30 21:18:17.677709983 (605h49m17.548793028s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.677718699 (605h49m17.548801745s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678074226 (605h49m17.549157271s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678083164 (605h49m17.549166209s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678238670 (605h49m17.549321715s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678248548 (605h49m17.549331593s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678269308 (605h49m17.549352343s) [seg_pos={0 0}, upload=530966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:18:17.678356864 (605h49m17.549439899s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:18:17.678364389 (605h49m17.549447424s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:18:17.965275506 (605h49m17.836358541s) [upload=530966]: Done uploading segments... Dec 30 21:18:17.965295543 (605h49m17.836378578s) [upload=530966]: Upload finished. err: <nil> Duration: 6.097365209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.......................................XX........X....X.X] duration: 30.00253274s trace: https://jaeger.dev.storj.io/trace/5dab66cd6a1c3e1f FAIL (2025-12-30 21:18:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ) context deadline exceeded ======LOGS====== Dec 30 21:18:15.038535656 (605h49m14.909618691s) [upload=530966]: Starting upload Dec 30 21:18:15.038593327 (605h49m14.909676362s) [upload=530966]: Done starting upload Dec 30 21:18:15.038772347 (605h49m14.909855382s) [upload=530966]: Uploading segments... Dec 30 21:18:15.038808045 (605h49m14.90989108s) [upload=530966]: Got next segment. Inline: false Dec 30 21:18:15.038836740 (605h49m14.909919785s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:18:15.038841619 (605h49m14.909924664s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:18:15.288031620 (605h49m15.159114665s) [seg_pos={0 0}, upload=530966]: Begin upload segment... Dec 30 21:18:15.288104969 (605h49m15.159188014s) [seg_pos={0 0}, upload=530966]: Done begin upload segment. Dec 30 21:18:15.288164493 (605h49m15.159247528s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288175323 (605h49m15.159258358s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288372088 (605h49m15.159455123s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.288771518 (605h49m15.159854553s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.289068294 (605h49m15.160151329s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.290630729 (605h49m15.161713794s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291035529 (605h49m15.162118584s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.291378322 (605h49m15.162461357s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.292676684 (605h49m15.163759729s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.294120654 (605h49m15.165203689s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.295450887 (605h49m15.166533922s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.297065291 (605h49m15.168148326s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298458484 (605h49m15.169541529s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.298531132 (605h49m15.169614177s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.299892464 (605h49m15.170975529s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:15.300767750 (605h49m15.171850796s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Uploading piece... Dec 30 21:18:16.997604903 (605h49m16.868687948s) [upload=530966]: Next returned nil segment Dec 30 21:18:16.997621605 (605h49m16.86870465s) [upload=530966]: Waiting for error group managing segments... Dec 30 21:18:17.275743075 (605h49m17.14682611s) [noise=true , address=145.79.227.3:4150 , piece_id=WAHJR7VQ2F27P7EQMMWWY55AQZKGPIBOQ4KW67PBZUOYOAHQMHXA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.280241811 (605h49m17.151324846s) [noise=true , address=145.79.227.9:4366 , piece_id=TXPTKEPON3YXUXUTUIF6AHGZHTZO2OQT6QDJ2RHZIL7JUCVANZZA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305053826 (605h49m17.176136861s) [noise=true , address=145.79.227.15:4875 , piece_id=FG7UYLVHG36WFGSCCWQLEEMGFMTYQFHCJ37K7AQU2H7UAHYKMMEA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.305171220 (605h49m17.176254265s) [noise=true , address=145.79.227.11:4729 , piece_id=YY6PRMGXOMCTDE5MKT3QOI4QUMJYWADHFQL6OYD2YJ65625CKBRA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.342799479 (605h49m17.213882514s) [noise=true , address=145.79.227.7:4294 , piece_id=ESTXC2ODDPEPEFDX7CEWANHSA2ECSJWDPGA3YJBIC35UBIMYOM4A, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.369239945 (605h49m17.24032299s) [noise=true , address=145.79.227.19:4078 , piece_id=JUKQMSGKTDTEHOUKU56FPAUNMIWI6RPVRUTC7VWQD4UX532LCLWQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.370450030 (605h49m17.241533075s) [noise=true , address=145.79.227.32:4836 , piece_id=N4YUKM4AA3FHT64EM7QOI7ZKTIP7FQB6TVIHMUCESVJRIISZYD4Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.546745219 (605h49m17.417828264s) [noise=true , address=77.90.31.16:4736 , piece_id=YBOJYXAVIJQICLKLDZ6QPWXHE5GNRF6BIJ3WZQMUUR6AEQOEOILQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.577326469 (605h49m17.448409504s) [noise=true , address=45.91.116.6:4976 , piece_id=GPTGRU2PLBYKGLLBW5GMOQA3X6UODVANOMJGITEEXU563DZA67CQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.579343340 (605h49m17.450426375s) [noise=true , address=77.90.31.19:4842 , piece_id=KZKZRAOHOJ4LL2RNNNBJNJPGLPTCFZADXLGLX5KV4QIVZWUO3VKQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.598317289 (605h49m17.469400345s) [noise=true , address=45.91.116.9:4264 , piece_id=FU5PF47LDWQOKJ2PFMGFRV6ULWFDFKGOZQU4YZJ4T54IV64FI2FQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.610488385 (605h49m17.48157143s) [noise=true , address=45.91.116.5:4299 , piece_id=Z6PKLXQHAMBSIYHKUOLWFTH3JZ6Q34B7OXWMFFBJ7UKWWGMHFF7Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677438386 (605h49m17.548521421s) [noise=true , address=45.91.116.13:4155 , piece_id=5OX5UBA2IE2F5IZBDDG4FDY2L6CSKMXWD2QWTWMBVSXGVKS2UQ3Q, seg_pos={0 0}, upload=530966]: Done uploading piece. err: <nil> Dec 30 21:18:17.677464406 (605h49m17.548547451s) [seg_pos={0 0}, upload=530966]: Segment reached optimal threshold of 13 pieces. Dec 30 21:18:17.677709983 (605h49m17.548793028s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.677718699 (605h49m17.548801745s) [noise=true , address=45.91.116.4:4940 , piece_id=LGUDGTNLFMOHE4RC7VLFYFPALUUZCYNTMG5Z3RWDUFDCZQRQSOFA, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678074226 (605h49m17.549157271s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678083164 (605h49m17.549166209s) [noise=true , address=145.79.227.33:4869 , piece_id=CDEQCFHZDYXIPBYEVJDM3BSPMS2OZZKBY7LZ6CONXPDW2JBE2JEQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678238670 (605h49m17.549321715s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 30 21:18:17.678248548 (605h49m17.549331593s) [noise=true , address=45.91.116.8:4045 , piece_id=FQ2V2CKF4VO54CQ4KNLQSSVP54JKLOIJZS4BIULXIMLBK4NCATPQ, seg_pos={0 0}, upload=530966]: Long tail cancellation due to optimal threshold Dec 30 21:18:17.678269308 (605h49m17.549352343s) [seg_pos={0 0}, upload=530966]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:18:17.678356864 (605h49m17.549439899s) [upload=530966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:18:17.678364389 (605h49m17.549447424s) [upload=530966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:18:17.965275506 (605h49m17.836358541s) [upload=530966]: Done uploading segments... Dec 30 21:18:17.965295543 (605h49m17.836378578s) [upload=530966]: Upload finished. err: <nil> Duration: 30.002558428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.......................................XX........X....X.] duration: 7.035405725s trace: https://jaeger.dev.storj.io/trace/2c6b5fc44fc48215 FAIL (2025-12-30 21:18:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:17:19.702452176 (605h48m19.573535211s) [upload=530953]: Starting upload Dec 30 21:17:19.702510217 (605h48m19.573593252s) [upload=530953]: Done starting upload Dec 30 21:17:19.702581172 (605h48m19.573664207s) [upload=530953]: Uploading segments... Dec 30 21:17:19.702680641 (605h48m19.573763676s) [upload=530953]: Got next segment. Inline: false Dec 30 21:17:19.702712381 (605h48m19.573795416s) [upload=530953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:17:19.702717851 (605h48m19.573800886s) [upload=530953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:17:19.952539186 (605h48m19.823622221s) [seg_pos={0 0}, upload=530953]: Begin upload segment... Dec 30 21:17:19.952594802 (605h48m19.823677837s) [seg_pos={0 0}, upload=530953]: Done begin upload segment. Dec 30 21:17:19.952659755 (605h48m19.82374279s) [noise=true , address=145.79.227.11:4730 , piece_id=R7ZIROA447ZL7KQDDLGI3BCZKT3EQYCX33BMIH2ZZ6OB637JAE7A, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.952695323 (605h48m19.823778368s) [noise=true , address=145.79.227.2:4113 , piece_id=THPIJF5XJXHIMTJMBR6AXCLYHEB6UX35YP73SSEHTO4XMFNGTYIQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.953142765 (605h48m19.82422581s) [noise=true , address=77.90.31.10:4518 , piece_id=3RJOBQ7ZGZDE2IOTIKSOYDPWZZEQGG3W5FVX5HR47XZTBCB5MALQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.953431024 (605h48m19.824514059s) [noise=true , address=145.79.227.45:4231 , piece_id=XPUMN6X4AROSGGLRJNJGHUAD7EKVPW4H56O4GJ42GAV6EQ37S3HQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.953705486 (605h48m19.824788521s) [noise=true , address=45.91.116.11:4119 , piece_id=MXKW3IE6HKF4NN6PM74ADT4FNQWPFRY6WLDKYPGO6QSSDL3A4ZGQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954004336 (605h48m19.825087371s) [noise=true , address=145.79.227.10:4403 , piece_id=2GCIEDWHNWKPQKDII6AB65GQI3ARXK4AQXFIGBCO4OXSEN4IHPJQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954276795 (605h48m19.82535983s) [noise=true , address=145.79.227.5:4222 , piece_id=5MV672UV34MWAJ4TCDQHVQ3ORDYXDZSRKNPK5SJYVY4KE6IS4DMQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954554834 (605h48m19.825637869s) [noise=true , address=145.79.227.15:4874 , piece_id=YGROH7IJMXIYC47AXMZOTH6YFMIF6PTRHB5B3GERT7SXUML6LU5A, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954874052 (605h48m19.825957087s) [noise=true , address=77.90.31.14:4663 , piece_id=F5VA26BYJS2U4MOUEHDV2J74HCTC3S3Q3UWZXA3XLNXYO7QRWTKQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.956198854 (605h48m19.827281889s) [noise=true , address=145.79.227.50:4375 , piece_id=IASC6OUH2RI23F6BAF2Z7M5DUUBNLSFFBXMH5NQ4EKFKKI62EKUQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.956322801 (605h48m19.827405836s) [noise=true , address=145.79.227.19:4077 , piece_id=N2X6B7DL6NHG3JSBB7WCRAAUXJTYDAZQJUWZ2ZQSEG65FUKAH3CQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.957460506 (605h48m19.828543541s) [noise=true , address=145.79.227.52:4446 , piece_id=IDTCIB2G5BMIUANGLJ355JTS4ETRTWXHG6AJ6BF7R445LTN7Y5OA, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.958814304 (605h48m19.829897339s) [noise=true , address=77.90.31.12:4590 , piece_id=B2QNXGNFFRP2YJ6JYBNDV3FMOKJUJDKU24VERKLFJ5RWDOAIMDVA, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.960182059 (605h48m19.831265094s) [noise=true , address=145.79.227.7:4295 , piece_id=QX7ROCQVHNJE5EA3OCBLARTVWVBYGQZ5NW7YA5HXMYNP46S5ACRQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.961585731 (605h48m19.832668766s) [noise=true , address=145.79.227.47:4267 , piece_id=R2ZO2BUT42EGG7G7FGEOYICXHTSNKW5V3YRIL4OG5HXIJXR5SABQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.962797909 (605h48m19.833880954s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5MYOZQSGXFP4SFD5V5TJN6OAWUALQT6BTMLCLUUIQQYJYWKXNYQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:21.607357509 (605h48m21.478440544s) [upload=530953]: Next returned nil segment Dec 30 21:17:21.607373280 (605h48m21.478456315s) [upload=530953]: Waiting for error group managing segments... Dec 30 21:17:21.955090849 (605h48m21.826173884s) [noise=true , address=145.79.227.50:4375 , piece_id=IASC6OUH2RI23F6BAF2Z7M5DUUBNLSFFBXMH5NQ4EKFKKI62EKUQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.955390059 (605h48m21.826473104s) [noise=true , address=145.79.227.5:4222 , piece_id=5MV672UV34MWAJ4TCDQHVQ3ORDYXDZSRKNPK5SJYVY4KE6IS4DMQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.955789249 (605h48m21.826872294s) [noise=true , address=145.79.227.45:4231 , piece_id=XPUMN6X4AROSGGLRJNJGHUAD7EKVPW4H56O4GJ42GAV6EQ37S3HQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.957300978 (605h48m21.828384023s) [noise=true , address=145.79.227.15:4874 , piece_id=YGROH7IJMXIYC47AXMZOTH6YFMIF6PTRHB5B3GERT7SXUML6LU5A, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.960726691 (605h48m21.831809726s) [noise=true , address=145.79.227.11:4730 , piece_id=R7ZIROA447ZL7KQDDLGI3BCZKT3EQYCX33BMIH2ZZ6OB637JAE7A, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.961406886 (605h48m21.832489922s) [noise=true , address=145.79.227.19:4077 , piece_id=N2X6B7DL6NHG3JSBB7WCRAAUXJTYDAZQJUWZ2ZQSEG65FUKAH3CQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.966415002 (605h48m21.837498037s) [noise=true , address=145.79.227.7:4295 , piece_id=QX7ROCQVHNJE5EA3OCBLARTVWVBYGQZ5NW7YA5HXMYNP46S5ACRQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.987700743 (605h48m21.858783788s) [noise=true , address=145.79.227.47:4267 , piece_id=R2ZO2BUT42EGG7G7FGEOYICXHTSNKW5V3YRIL4OG5HXIJXR5SABQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.991724715 (605h48m21.86280775s) [noise=true , address=145.79.227.10:4403 , piece_id=2GCIEDWHNWKPQKDII6AB65GQI3ARXK4AQXFIGBCO4OXSEN4IHPJQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.028289419 (605h48m21.899372454s) [noise=true , address=145.79.227.52:4446 , piece_id=IDTCIB2G5BMIUANGLJ355JTS4ETRTWXHG6AJ6BF7R445LTN7Y5OA, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.045253341 (605h48m21.916336376s) [noise=true , address=145.79.227.2:4113 , piece_id=THPIJF5XJXHIMTJMBR6AXCLYHEB6UX35YP73SSEHTO4XMFNGTYIQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.209035293 (605h48m22.080118328s) [noise=true , address=77.90.31.12:4590 , piece_id=B2QNXGNFFRP2YJ6JYBNDV3FMOKJUJDKU24VERKLFJ5RWDOAIMDVA, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.215139276 (605h48m22.086222321s) [noise=true , address=77.90.31.14:4663 , piece_id=F5VA26BYJS2U4MOUEHDV2J74HCTC3S3Q3UWZXA3XLNXYO7QRWTKQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.215161318 (605h48m22.086244363s) [seg_pos={0 0}, upload=530953]: Segment reached optimal threshold of 13 pieces. Dec 30 21:17:22.215406164 (605h48m22.086489209s) [noise=true , address=77.90.31.10:4518 , piece_id=3RJOBQ7ZGZDE2IOTIKSOYDPWZZEQGG3W5FVX5HR47XZTBCB5MALQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 21:17:22.215414220 (605h48m22.086497265s) [noise=true , address=77.90.31.10:4518 , piece_id=3RJOBQ7ZGZDE2IOTIKSOYDPWZZEQGG3W5FVX5HR47XZTBCB5MALQ, seg_pos={0 0}, upload=530953]: Long tail cancellation due to optimal threshold Dec 30 21:17:22.215423016 (605h48m22.086506051s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5MYOZQSGXFP4SFD5V5TJN6OAWUALQT6BTMLCLUUIQQYJYWKXNYQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 21:17:22.215431141 (605h48m22.086514176s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5MYOZQSGXFP4SFD5V5TJN6OAWUALQT6BTMLCLUUIQQYJYWKXNYQ, seg_pos={0 0}, upload=530953]: Long tail cancellation due to optimal threshold Dec 30 21:17:22.215622346 (605h48m22.086705381s) [noise=true , address=45.91.116.11:4119 , piece_id=MXKW3IE6HKF4NN6PM74ADT4FNQWPFRY6WLDKYPGO6QSSDL3A4ZGQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 21:17:22.215642714 (605h48m22.086725749s) [noise=true , address=45.91.116.11:4119 , piece_id=MXKW3IE6HKF4NN6PM74ADT4FNQWPFRY6WLDKYPGO6QSSDL3A4ZGQ, seg_pos={0 0}, upload=530953]: Long tail cancellation due to optimal threshold Dec 30 21:17:22.215658575 (605h48m22.08674162s) [seg_pos={0 0}, upload=530953]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:17:22.215739248 (605h48m22.086822283s) [upload=530953]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:17:22.215744989 (605h48m22.086828024s) [upload=530953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:17:22.520150817 (605h48m22.391233852s) [upload=530953]: Done uploading segments... Dec 30 21:17:22.520175583 (605h48m22.391258618s) [upload=530953]: Upload finished. err: <nil> Duration: 7.035436103s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X......X.........................................] duration: 3.581685103s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:18:01 UTC) orbiter warning: context deadline exceeded Duration: 3.581711223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.......................................XX........X....X] duration: 30.00296653s trace: https://jaeger.dev.storj.io/trace/2c6b5fc44fc48215 FAIL (2025-12-30 21:17:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 21:17:19.702452176 (605h48m19.573535211s) [upload=530953]: Starting upload Dec 30 21:17:19.702510217 (605h48m19.573593252s) [upload=530953]: Done starting upload Dec 30 21:17:19.702581172 (605h48m19.573664207s) [upload=530953]: Uploading segments... Dec 30 21:17:19.702680641 (605h48m19.573763676s) [upload=530953]: Got next segment. Inline: false Dec 30 21:17:19.702712381 (605h48m19.573795416s) [upload=530953]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:17:19.702717851 (605h48m19.573800886s) [upload=530953]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:17:19.952539186 (605h48m19.823622221s) [seg_pos={0 0}, upload=530953]: Begin upload segment... Dec 30 21:17:19.952594802 (605h48m19.823677837s) [seg_pos={0 0}, upload=530953]: Done begin upload segment. Dec 30 21:17:19.952659755 (605h48m19.82374279s) [noise=true , address=145.79.227.11:4730 , piece_id=R7ZIROA447ZL7KQDDLGI3BCZKT3EQYCX33BMIH2ZZ6OB637JAE7A, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.952695323 (605h48m19.823778368s) [noise=true , address=145.79.227.2:4113 , piece_id=THPIJF5XJXHIMTJMBR6AXCLYHEB6UX35YP73SSEHTO4XMFNGTYIQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.953142765 (605h48m19.82422581s) [noise=true , address=77.90.31.10:4518 , piece_id=3RJOBQ7ZGZDE2IOTIKSOYDPWZZEQGG3W5FVX5HR47XZTBCB5MALQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.953431024 (605h48m19.824514059s) [noise=true , address=145.79.227.45:4231 , piece_id=XPUMN6X4AROSGGLRJNJGHUAD7EKVPW4H56O4GJ42GAV6EQ37S3HQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.953705486 (605h48m19.824788521s) [noise=true , address=45.91.116.11:4119 , piece_id=MXKW3IE6HKF4NN6PM74ADT4FNQWPFRY6WLDKYPGO6QSSDL3A4ZGQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954004336 (605h48m19.825087371s) [noise=true , address=145.79.227.10:4403 , piece_id=2GCIEDWHNWKPQKDII6AB65GQI3ARXK4AQXFIGBCO4OXSEN4IHPJQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954276795 (605h48m19.82535983s) [noise=true , address=145.79.227.5:4222 , piece_id=5MV672UV34MWAJ4TCDQHVQ3ORDYXDZSRKNPK5SJYVY4KE6IS4DMQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954554834 (605h48m19.825637869s) [noise=true , address=145.79.227.15:4874 , piece_id=YGROH7IJMXIYC47AXMZOTH6YFMIF6PTRHB5B3GERT7SXUML6LU5A, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.954874052 (605h48m19.825957087s) [noise=true , address=77.90.31.14:4663 , piece_id=F5VA26BYJS2U4MOUEHDV2J74HCTC3S3Q3UWZXA3XLNXYO7QRWTKQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.956198854 (605h48m19.827281889s) [noise=true , address=145.79.227.50:4375 , piece_id=IASC6OUH2RI23F6BAF2Z7M5DUUBNLSFFBXMH5NQ4EKFKKI62EKUQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.956322801 (605h48m19.827405836s) [noise=true , address=145.79.227.19:4077 , piece_id=N2X6B7DL6NHG3JSBB7WCRAAUXJTYDAZQJUWZ2ZQSEG65FUKAH3CQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.957460506 (605h48m19.828543541s) [noise=true , address=145.79.227.52:4446 , piece_id=IDTCIB2G5BMIUANGLJ355JTS4ETRTWXHG6AJ6BF7R445LTN7Y5OA, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.958814304 (605h48m19.829897339s) [noise=true , address=77.90.31.12:4590 , piece_id=B2QNXGNFFRP2YJ6JYBNDV3FMOKJUJDKU24VERKLFJ5RWDOAIMDVA, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.960182059 (605h48m19.831265094s) [noise=true , address=145.79.227.7:4295 , piece_id=QX7ROCQVHNJE5EA3OCBLARTVWVBYGQZ5NW7YA5HXMYNP46S5ACRQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.961585731 (605h48m19.832668766s) [noise=true , address=145.79.227.47:4267 , piece_id=R2ZO2BUT42EGG7G7FGEOYICXHTSNKW5V3YRIL4OG5HXIJXR5SABQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:19.962797909 (605h48m19.833880954s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5MYOZQSGXFP4SFD5V5TJN6OAWUALQT6BTMLCLUUIQQYJYWKXNYQ, seg_pos={0 0}, upload=530953]: Uploading piece... Dec 30 21:17:21.607357509 (605h48m21.478440544s) [upload=530953]: Next returned nil segment Dec 30 21:17:21.607373280 (605h48m21.478456315s) [upload=530953]: Waiting for error group managing segments... Dec 30 21:17:21.955090849 (605h48m21.826173884s) [noise=true , address=145.79.227.50:4375 , piece_id=IASC6OUH2RI23F6BAF2Z7M5DUUBNLSFFBXMH5NQ4EKFKKI62EKUQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.955390059 (605h48m21.826473104s) [noise=true , address=145.79.227.5:4222 , piece_id=5MV672UV34MWAJ4TCDQHVQ3ORDYXDZSRKNPK5SJYVY4KE6IS4DMQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.955789249 (605h48m21.826872294s) [noise=true , address=145.79.227.45:4231 , piece_id=XPUMN6X4AROSGGLRJNJGHUAD7EKVPW4H56O4GJ42GAV6EQ37S3HQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.957300978 (605h48m21.828384023s) [noise=true , address=145.79.227.15:4874 , piece_id=YGROH7IJMXIYC47AXMZOTH6YFMIF6PTRHB5B3GERT7SXUML6LU5A, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.960726691 (605h48m21.831809726s) [noise=true , address=145.79.227.11:4730 , piece_id=R7ZIROA447ZL7KQDDLGI3BCZKT3EQYCX33BMIH2ZZ6OB637JAE7A, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.961406886 (605h48m21.832489922s) [noise=true , address=145.79.227.19:4077 , piece_id=N2X6B7DL6NHG3JSBB7WCRAAUXJTYDAZQJUWZ2ZQSEG65FUKAH3CQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.966415002 (605h48m21.837498037s) [noise=true , address=145.79.227.7:4295 , piece_id=QX7ROCQVHNJE5EA3OCBLARTVWVBYGQZ5NW7YA5HXMYNP46S5ACRQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.987700743 (605h48m21.858783788s) [noise=true , address=145.79.227.47:4267 , piece_id=R2ZO2BUT42EGG7G7FGEOYICXHTSNKW5V3YRIL4OG5HXIJXR5SABQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:21.991724715 (605h48m21.86280775s) [noise=true , address=145.79.227.10:4403 , piece_id=2GCIEDWHNWKPQKDII6AB65GQI3ARXK4AQXFIGBCO4OXSEN4IHPJQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.028289419 (605h48m21.899372454s) [noise=true , address=145.79.227.52:4446 , piece_id=IDTCIB2G5BMIUANGLJ355JTS4ETRTWXHG6AJ6BF7R445LTN7Y5OA, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.045253341 (605h48m21.916336376s) [noise=true , address=145.79.227.2:4113 , piece_id=THPIJF5XJXHIMTJMBR6AXCLYHEB6UX35YP73SSEHTO4XMFNGTYIQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.209035293 (605h48m22.080118328s) [noise=true , address=77.90.31.12:4590 , piece_id=B2QNXGNFFRP2YJ6JYBNDV3FMOKJUJDKU24VERKLFJ5RWDOAIMDVA, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.215139276 (605h48m22.086222321s) [noise=true , address=77.90.31.14:4663 , piece_id=F5VA26BYJS2U4MOUEHDV2J74HCTC3S3Q3UWZXA3XLNXYO7QRWTKQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: <nil> Dec 30 21:17:22.215161318 (605h48m22.086244363s) [seg_pos={0 0}, upload=530953]: Segment reached optimal threshold of 13 pieces. Dec 30 21:17:22.215406164 (605h48m22.086489209s) [noise=true , address=77.90.31.10:4518 , piece_id=3RJOBQ7ZGZDE2IOTIKSOYDPWZZEQGG3W5FVX5HR47XZTBCB5MALQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 21:17:22.215414220 (605h48m22.086497265s) [noise=true , address=77.90.31.10:4518 , piece_id=3RJOBQ7ZGZDE2IOTIKSOYDPWZZEQGG3W5FVX5HR47XZTBCB5MALQ, seg_pos={0 0}, upload=530953]: Long tail cancellation due to optimal threshold Dec 30 21:17:22.215423016 (605h48m22.086506051s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5MYOZQSGXFP4SFD5V5TJN6OAWUALQT6BTMLCLUUIQQYJYWKXNYQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 30 21:17:22.215431141 (605h48m22.086514176s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5MYOZQSGXFP4SFD5V5TJN6OAWUALQT6BTMLCLUUIQQYJYWKXNYQ, seg_pos={0 0}, upload=530953]: Long tail cancellation due to optimal threshold Dec 30 21:17:22.215622346 (605h48m22.086705381s) [noise=true , address=45.91.116.11:4119 , piece_id=MXKW3IE6HKF4NN6PM74ADT4FNQWPFRY6WLDKYPGO6QSSDL3A4ZGQ, seg_pos={0 0}, upload=530953]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 21:17:22.215642714 (605h48m22.086725749s) [noise=true , address=45.91.116.11:4119 , piece_id=MXKW3IE6HKF4NN6PM74ADT4FNQWPFRY6WLDKYPGO6QSSDL3A4ZGQ, seg_pos={0 0}, upload=530953]: Long tail cancellation due to optimal threshold Dec 30 21:17:22.215658575 (605h48m22.08674162s) [seg_pos={0 0}, upload=530953]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:17:22.215739248 (605h48m22.086822283s) [upload=530953]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:17:22.215744989 (605h48m22.086828024s) [upload=530953]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:17:22.520150817 (605h48m22.391233852s) [upload=530953]: Done uploading segments... Dec 30 21:17:22.520175583 (605h48m22.391258618s) [upload=530953]: Upload finished. err: <nil> Duration: 30.00299331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.......................................XX........X....] duration: 26.848150067s trace: https://jaeger.dev.storj.io/trace/7b93f843c10b3452 FAIL (2025-12-30 21:17:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ) context deadline exceeded ======LOGS====== Dec 30 21:14:12.787614246 (605h45m12.658697322s) [upload=530904]: Starting upload Dec 30 21:14:12.787786875 (605h45m12.65886991s) [upload=530904]: Done starting upload Dec 30 21:14:12.787902596 (605h45m12.658985631s) [upload=530904]: Uploading segments... Dec 30 21:14:12.788311464 (605h45m12.659394509s) [upload=530904]: Got next segment. Inline: false Dec 30 21:14:12.788356711 (605h45m12.659439756s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:14:12.788366028 (605h45m12.659449063s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:14:13.030735302 (605h45m12.901818337s) [seg_pos={0 0}, upload=530904]: Begin upload segment... Dec 30 21:14:13.030785527 (605h45m12.901868562s) [seg_pos={0 0}, upload=530904]: Done begin upload segment. Dec 30 21:14:13.030829911 (605h45m12.901912946s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.030846614 (605h45m12.901929659s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031358858 (605h45m12.902441893s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031649803 (605h45m12.902732848s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032065074 (605h45m12.903148119s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032450367 (605h45m12.903533412s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032843005 (605h45m12.90392604s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033244579 (605h45m12.904327614s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033592822 (605h45m12.904675857s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033951656 (605h45m12.905034691s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034083316 (605h45m12.905166361s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034511221 (605h45m12.905594266s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034840318 (605h45m12.905923353s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.035227645 (605h45m12.90631068s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.036162956 (605h45m12.907245991s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.037246129 (605h45m12.908329164s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:14.727982425 (605h45m14.59906546s) [upload=530904]: Next returned nil segment Dec 30 21:14:14.727996582 (605h45m14.599079627s) [upload=530904]: Waiting for error group managing segments... Dec 30 21:14:15.004044033 (605h45m14.875127078s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.013462807 (605h45m14.884545852s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.038808499 (605h45m14.909891544s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039219421 (605h45m14.910302456s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039702772 (605h45m14.910785817s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045537872 (605h45m14.916620917s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045770185 (605h45m14.91685322s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.058746984 (605h45m14.929830029s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.062368871 (605h45m14.933451906s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.342184608 (605h45m15.213267653s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.429253157 (605h45m15.300336192s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.966285899 (605h45m15.837368934s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069945692 (605h45m15.941028727s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069971030 (605h45m15.941054075s) [seg_pos={0 0}, upload=530904]: Segment reached optimal threshold of 13 pieces. Dec 30 21:14:16.070236966 (605h45m15.941320021s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070246173 (605h45m15.941329208s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070401289 (605h45m15.941484324s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070435033 (605h45m15.941518078s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070574399 (605h45m15.941657454s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070584387 (605h45m15.941667433s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070607582 (605h45m15.941690617s) [seg_pos={0 0}, upload=530904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:14:16.070695419 (605h45m15.941778464s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:14:16.070701932 (605h45m15.941784977s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:14:16.368603977 (605h45m16.239687022s) [upload=530904]: Done uploading segments... Dec 30 21:14:16.368624797 (605h45m16.239707832s) [upload=530904]: Upload finished. err: <nil> Duration: 26.848170266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X......X........................................] duration: 3.623467351s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:17:06 UTC) orbiter warning: context deadline exceeded Duration: 3.623526424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.....X.......................................XX........X...] duration: 7.045196084s trace: https://jaeger.dev.storj.io/trace/7b93f843c10b3452 FAIL (2025-12-30 21:16:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ) context deadline exceeded ======LOGS====== Dec 30 21:14:12.787614246 (605h45m12.658697322s) [upload=530904]: Starting upload Dec 30 21:14:12.787786875 (605h45m12.65886991s) [upload=530904]: Done starting upload Dec 30 21:14:12.787902596 (605h45m12.658985631s) [upload=530904]: Uploading segments... Dec 30 21:14:12.788311464 (605h45m12.659394509s) [upload=530904]: Got next segment. Inline: false Dec 30 21:14:12.788356711 (605h45m12.659439756s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:14:12.788366028 (605h45m12.659449063s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:14:13.030735302 (605h45m12.901818337s) [seg_pos={0 0}, upload=530904]: Begin upload segment... Dec 30 21:14:13.030785527 (605h45m12.901868562s) [seg_pos={0 0}, upload=530904]: Done begin upload segment. Dec 30 21:14:13.030829911 (605h45m12.901912946s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.030846614 (605h45m12.901929659s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031358858 (605h45m12.902441893s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031649803 (605h45m12.902732848s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032065074 (605h45m12.903148119s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032450367 (605h45m12.903533412s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032843005 (605h45m12.90392604s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033244579 (605h45m12.904327614s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033592822 (605h45m12.904675857s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033951656 (605h45m12.905034691s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034083316 (605h45m12.905166361s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034511221 (605h45m12.905594266s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034840318 (605h45m12.905923353s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.035227645 (605h45m12.90631068s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.036162956 (605h45m12.907245991s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.037246129 (605h45m12.908329164s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:14.727982425 (605h45m14.59906546s) [upload=530904]: Next returned nil segment Dec 30 21:14:14.727996582 (605h45m14.599079627s) [upload=530904]: Waiting for error group managing segments... Dec 30 21:14:15.004044033 (605h45m14.875127078s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.013462807 (605h45m14.884545852s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.038808499 (605h45m14.909891544s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039219421 (605h45m14.910302456s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039702772 (605h45m14.910785817s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045537872 (605h45m14.916620917s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045770185 (605h45m14.91685322s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.058746984 (605h45m14.929830029s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.062368871 (605h45m14.933451906s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.342184608 (605h45m15.213267653s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.429253157 (605h45m15.300336192s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.966285899 (605h45m15.837368934s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069945692 (605h45m15.941028727s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069971030 (605h45m15.941054075s) [seg_pos={0 0}, upload=530904]: Segment reached optimal threshold of 13 pieces. Dec 30 21:14:16.070236966 (605h45m15.941320021s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070246173 (605h45m15.941329208s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070401289 (605h45m15.941484324s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070435033 (605h45m15.941518078s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070574399 (605h45m15.941657454s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070584387 (605h45m15.941667433s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070607582 (605h45m15.941690617s) [seg_pos={0 0}, upload=530904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:14:16.070695419 (605h45m15.941778464s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:14:16.070701932 (605h45m15.941784977s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:14:16.368603977 (605h45m16.239687022s) [upload=530904]: Done uploading segments... Dec 30 21:14:16.368624797 (605h45m16.239707832s) [upload=530904]: Upload finished. err: <nil> Duration: 7.045224069s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X......X.......................................] duration: 3.67112653s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:16:13 UTC) orbiter warning: context deadline exceeded Duration: 3.671152058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....X.......................................XX........X..] duration: 5.957295087s trace: https://jaeger.dev.storj.io/trace/7b93f843c10b3452 FAIL (2025-12-30 21:15:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ) context deadline exceeded ======LOGS====== Dec 30 21:14:12.787614246 (605h45m12.658697322s) [upload=530904]: Starting upload Dec 30 21:14:12.787786875 (605h45m12.65886991s) [upload=530904]: Done starting upload Dec 30 21:14:12.787902596 (605h45m12.658985631s) [upload=530904]: Uploading segments... Dec 30 21:14:12.788311464 (605h45m12.659394509s) [upload=530904]: Got next segment. Inline: false Dec 30 21:14:12.788356711 (605h45m12.659439756s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:14:12.788366028 (605h45m12.659449063s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:14:13.030735302 (605h45m12.901818337s) [seg_pos={0 0}, upload=530904]: Begin upload segment... Dec 30 21:14:13.030785527 (605h45m12.901868562s) [seg_pos={0 0}, upload=530904]: Done begin upload segment. Dec 30 21:14:13.030829911 (605h45m12.901912946s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.030846614 (605h45m12.901929659s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031358858 (605h45m12.902441893s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031649803 (605h45m12.902732848s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032065074 (605h45m12.903148119s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032450367 (605h45m12.903533412s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032843005 (605h45m12.90392604s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033244579 (605h45m12.904327614s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033592822 (605h45m12.904675857s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033951656 (605h45m12.905034691s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034083316 (605h45m12.905166361s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034511221 (605h45m12.905594266s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034840318 (605h45m12.905923353s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.035227645 (605h45m12.90631068s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.036162956 (605h45m12.907245991s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.037246129 (605h45m12.908329164s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:14.727982425 (605h45m14.59906546s) [upload=530904]: Next returned nil segment Dec 30 21:14:14.727996582 (605h45m14.599079627s) [upload=530904]: Waiting for error group managing segments... Dec 30 21:14:15.004044033 (605h45m14.875127078s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.013462807 (605h45m14.884545852s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.038808499 (605h45m14.909891544s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039219421 (605h45m14.910302456s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039702772 (605h45m14.910785817s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045537872 (605h45m14.916620917s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045770185 (605h45m14.91685322s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.058746984 (605h45m14.929830029s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.062368871 (605h45m14.933451906s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.342184608 (605h45m15.213267653s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.429253157 (605h45m15.300336192s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.966285899 (605h45m15.837368934s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069945692 (605h45m15.941028727s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069971030 (605h45m15.941054075s) [seg_pos={0 0}, upload=530904]: Segment reached optimal threshold of 13 pieces. Dec 30 21:14:16.070236966 (605h45m15.941320021s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070246173 (605h45m15.941329208s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070401289 (605h45m15.941484324s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070435033 (605h45m15.941518078s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070574399 (605h45m15.941657454s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070584387 (605h45m15.941667433s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070607582 (605h45m15.941690617s) [seg_pos={0 0}, upload=530904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:14:16.070695419 (605h45m15.941778464s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:14:16.070701932 (605h45m15.941784977s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:14:16.368603977 (605h45m16.239687022s) [upload=530904]: Done uploading segments... Dec 30 21:14:16.368624797 (605h45m16.239707832s) [upload=530904]: Upload finished. err: <nil> Duration: 5.957323672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X.....X.......................................XX........X.] duration: 14.097407272s trace: https://jaeger.dev.storj.io/trace/7b93f843c10b3452 FAIL (2025-12-30 21:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ) context deadline exceeded ======LOGS====== Dec 30 21:14:12.787614246 (605h45m12.658697322s) [upload=530904]: Starting upload Dec 30 21:14:12.787786875 (605h45m12.65886991s) [upload=530904]: Done starting upload Dec 30 21:14:12.787902596 (605h45m12.658985631s) [upload=530904]: Uploading segments... Dec 30 21:14:12.788311464 (605h45m12.659394509s) [upload=530904]: Got next segment. Inline: false Dec 30 21:14:12.788356711 (605h45m12.659439756s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:14:12.788366028 (605h45m12.659449063s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:14:13.030735302 (605h45m12.901818337s) [seg_pos={0 0}, upload=530904]: Begin upload segment... Dec 30 21:14:13.030785527 (605h45m12.901868562s) [seg_pos={0 0}, upload=530904]: Done begin upload segment. Dec 30 21:14:13.030829911 (605h45m12.901912946s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.030846614 (605h45m12.901929659s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031358858 (605h45m12.902441893s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031649803 (605h45m12.902732848s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032065074 (605h45m12.903148119s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032450367 (605h45m12.903533412s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032843005 (605h45m12.90392604s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033244579 (605h45m12.904327614s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033592822 (605h45m12.904675857s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033951656 (605h45m12.905034691s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034083316 (605h45m12.905166361s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034511221 (605h45m12.905594266s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034840318 (605h45m12.905923353s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.035227645 (605h45m12.90631068s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.036162956 (605h45m12.907245991s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.037246129 (605h45m12.908329164s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:14.727982425 (605h45m14.59906546s) [upload=530904]: Next returned nil segment Dec 30 21:14:14.727996582 (605h45m14.599079627s) [upload=530904]: Waiting for error group managing segments... Dec 30 21:14:15.004044033 (605h45m14.875127078s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.013462807 (605h45m14.884545852s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.038808499 (605h45m14.909891544s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039219421 (605h45m14.910302456s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039702772 (605h45m14.910785817s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045537872 (605h45m14.916620917s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045770185 (605h45m14.91685322s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.058746984 (605h45m14.929830029s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.062368871 (605h45m14.933451906s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.342184608 (605h45m15.213267653s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.429253157 (605h45m15.300336192s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.966285899 (605h45m15.837368934s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069945692 (605h45m15.941028727s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069971030 (605h45m15.941054075s) [seg_pos={0 0}, upload=530904]: Segment reached optimal threshold of 13 pieces. Dec 30 21:14:16.070236966 (605h45m15.941320021s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070246173 (605h45m15.941329208s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070401289 (605h45m15.941484324s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070435033 (605h45m15.941518078s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070574399 (605h45m15.941657454s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070584387 (605h45m15.941667433s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070607582 (605h45m15.941690617s) [seg_pos={0 0}, upload=530904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:14:16.070695419 (605h45m15.941778464s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:14:16.070701932 (605h45m15.941784977s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:14:16.368603977 (605h45m16.239687022s) [upload=530904]: Done uploading segments... Dec 30 21:14:16.368624797 (605h45m16.239707832s) [upload=530904]: Upload finished. err: <nil> Duration: 14.097488588s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X......X......................................] duration: 3.627864672s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:14:59 UTC) orbiter warning: context deadline exceeded Duration: 3.627901082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.X.....X.......................................XX........X] duration: 30.002522387s trace: https://jaeger.dev.storj.io/trace/7b93f843c10b3452 FAIL (2025-12-30 21:14:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ) context deadline exceeded ======LOGS====== Dec 30 21:14:12.787614246 (605h45m12.658697322s) [upload=530904]: Starting upload Dec 30 21:14:12.787786875 (605h45m12.65886991s) [upload=530904]: Done starting upload Dec 30 21:14:12.787902596 (605h45m12.658985631s) [upload=530904]: Uploading segments... Dec 30 21:14:12.788311464 (605h45m12.659394509s) [upload=530904]: Got next segment. Inline: false Dec 30 21:14:12.788356711 (605h45m12.659439756s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:14:12.788366028 (605h45m12.659449063s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:14:13.030735302 (605h45m12.901818337s) [seg_pos={0 0}, upload=530904]: Begin upload segment... Dec 30 21:14:13.030785527 (605h45m12.901868562s) [seg_pos={0 0}, upload=530904]: Done begin upload segment. Dec 30 21:14:13.030829911 (605h45m12.901912946s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.030846614 (605h45m12.901929659s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031358858 (605h45m12.902441893s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.031649803 (605h45m12.902732848s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032065074 (605h45m12.903148119s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032450367 (605h45m12.903533412s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.032843005 (605h45m12.90392604s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033244579 (605h45m12.904327614s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033592822 (605h45m12.904675857s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.033951656 (605h45m12.905034691s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034083316 (605h45m12.905166361s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034511221 (605h45m12.905594266s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.034840318 (605h45m12.905923353s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.035227645 (605h45m12.90631068s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.036162956 (605h45m12.907245991s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:13.037246129 (605h45m12.908329164s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Uploading piece... Dec 30 21:14:14.727982425 (605h45m14.59906546s) [upload=530904]: Next returned nil segment Dec 30 21:14:14.727996582 (605h45m14.599079627s) [upload=530904]: Waiting for error group managing segments... Dec 30 21:14:15.004044033 (605h45m14.875127078s) [noise=true , address=145.79.227.41:4087 , piece_id=42NP3LDOUSXXIZOQT6B5QJPD2JKDPGSKJVFQSOGY3WXSXZLXYL4A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.013462807 (605h45m14.884545852s) [noise=true , address=145.79.227.45:4231 , piece_id=6GQLT3WILAOGCIEJXJDEM245SRDI622CP4POYOUSAHT63GEIJSQQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.038808499 (605h45m14.909891544s) [noise=true , address=145.79.227.14:4837 , piece_id=2U2WZLM7SZ4NCI7LDHODPH26IEHQ6XDGXWPZ3P24LONHNOL2PPQA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039219421 (605h45m14.910302456s) [noise=true , address=145.79.227.27:4691 , piece_id=3EAH3TFSSRZXTO2ZM4WSGFAN7RAH5V3BYXHPEEA3O446YVWSCSBA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.039702772 (605h45m14.910785817s) [noise=true , address=145.79.227.36:4519 , piece_id=XKGVRRPHDTR7HFQLF3GP33XOFL443WRMYV2DIB4BLP2T73VLFBWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045537872 (605h45m14.916620917s) [noise=true , address=145.79.227.51:4410 , piece_id=OVMVMLEI53PFKN7WRUXOIDXFIT5CQESPJPKC5ZCMGZF4ADFOM2OA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.045770185 (605h45m14.91685322s) [noise=true , address=145.79.227.11:4730 , piece_id=OJRJOICV3NWUYLUTRGTRVXJAQVQFJUDTGOUKV3PZT5DB4IERVKLA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.058746984 (605h45m14.929830029s) [noise=true , address=145.79.227.15:4876 , piece_id=FSZ3ZM7SHD7S3MSUFNXNWO4KOL7PJJILD3VCP3S3RNTKN5U7WQWA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.062368871 (605h45m14.933451906s) [noise=true , address=145.79.227.52:4446 , piece_id=E2WT4KAEEXEP7H2WJAZ74ZAZJJSZW3T7EA3VLMA2FJH36F3MXB6Q, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.342184608 (605h45m15.213267653s) [noise=true , address=45.91.116.6:4974 , piece_id=XKPVOYHFSD6HZRPYRMHH5LQDMP4DMWPTK32AAF4JMAEXUIN2QCUQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.429253157 (605h45m15.300336192s) [noise=true , address=45.91.116.9:4262 , piece_id=S3G5WJIRV7NPXFAZN4I45JJU3BJBHDWG3UICIRBUZ5YZF4Y3IM5A, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:15.966285899 (605h45m15.837368934s) [noise=true , address=77.90.31.12:4590 , piece_id=BM2KKBM65ZAIBARGRG5XWFSF7RBFT57ZE7IO2V67X7O55TFJOHFA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069945692 (605h45m15.941028727s) [noise=true , address=45.91.116.7:4010 , piece_id=DGVKLUR7BOJ2G7AWJ6JW5L2APPAWQBON4MK2HUQOZGIMH6J4A7DQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: <nil> Dec 30 21:14:16.069971030 (605h45m15.941054075s) [seg_pos={0 0}, upload=530904]: Segment reached optimal threshold of 13 pieces. Dec 30 21:14:16.070236966 (605h45m15.941320021s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070246173 (605h45m15.941329208s) [noise=true , address=145.79.227.40:4051 , piece_id=RHU6FCNSSNUQIW4MX7O2LT3HM3DII6HX6Y4OYOZSVAE6BF7R2AOA, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070401289 (605h45m15.941484324s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070435033 (605h45m15.941518078s) [noise=true , address=145.79.227.30:4762 , piece_id=XD2DETJP66JDP6FLBOBINVUEPV6VMWHCUN67RR2BRM6ZYFPGQKTQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070574399 (605h45m15.941657454s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 30 21:14:16.070584387 (605h45m15.941667433s) [noise=true , address=45.91.116.11:4118 , piece_id=UL73FFIXCL4RDOB4WFY3R6ZHBY64E6ZSNIJZUKQK52SKODWQ3DKQ, seg_pos={0 0}, upload=530904]: Long tail cancellation due to optimal threshold Dec 30 21:14:16.070607582 (605h45m15.941690617s) [seg_pos={0 0}, upload=530904]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:14:16.070695419 (605h45m15.941778464s) [upload=530904]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:14:16.070701932 (605h45m15.941784977s) [upload=530904]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:14:16.368603977 (605h45m16.239687022s) [upload=530904]: Done uploading segments... Dec 30 21:14:16.368624797 (605h45m16.239707832s) [upload=530904]: Upload finished. err: <nil> Duration: 30.002536854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X.....X.......................................XX........] duration: 26.283941595s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:13:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 26.283969057s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X......X.....................................] duration: 3.599399484s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:13:48 UTC) orbiter warning: context deadline exceeded Duration: 3.599425022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X.....X.......................................XX.......] duration: 27.932492494s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:13:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 27.932520449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X......X....................................] duration: 3.672647068s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:12:59 UTC) orbiter warning: context deadline exceeded Duration: 3.672671946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X.....X.......................................XX......] duration: 6.198454054s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:12:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 6.198479923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X......X...................................] duration: 3.525254854s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:11:47 UTC) orbiter warning: context deadline exceeded Duration: 3.52529494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X.....X.......................................XX.....] duration: 10.767892845s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:11:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 10.767923332s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X......X..................................] duration: 3.607254137s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:10:52 UTC) orbiter warning: context deadline exceeded Duration: 3.607278232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.X.....X.......................................XX....] duration: 5.9427518s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:10:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 5.942782018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X......X.................................] duration: 3.678196469s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:09:48 UTC) orbiter warning: context deadline exceeded Duration: 3.678526929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.X.....X.......................................XX...] duration: 6.144789876s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 6.144819141s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X......X................................] duration: 3.759530891s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:08:59 UTC) orbiter warning: context deadline exceeded Duration: 3.759556972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.......X.X.....X.......................................XX..] duration: 6.258589241s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:08:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 6.258616142s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X......X...............................] duration: 3.531315263s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:08:06 UTC) orbiter warning: context deadline exceeded Duration: 3.531342245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.......X.X.....X.......................................XX.] duration: 27.39644523s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:07:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 27.396469697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.......X.X.....X.......................................XX] duration: 30.004014032s trace: https://jaeger.dev.storj.io/trace/3ec355fd1bf50d73 FAIL (2025-12-30 21:07:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ) context deadline exceeded ======LOGS====== Dec 30 21:06:44.393769829 (605h37m44.264852874s) [upload=530800]: Starting upload Dec 30 21:06:44.393854571 (605h37m44.264937606s) [upload=530800]: Done starting upload Dec 30 21:06:44.393960132 (605h37m44.265043177s) [upload=530800]: Uploading segments... Dec 30 21:06:44.394059971 (605h37m44.265143016s) [upload=530800]: Got next segment. Inline: false Dec 30 21:06:44.394125175 (605h37m44.26520822s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 21:06:44.394132098 (605h37m44.265215143s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:06:44.664457549 (605h37m44.535540614s) [seg_pos={0 0}, upload=530800]: Begin upload segment... Dec 30 21:06:44.664514428 (605h37m44.535597463s) [seg_pos={0 0}, upload=530800]: Done begin upload segment. Dec 30 21:06:44.664549484 (605h37m44.535632519s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.664554564 (605h37m44.535637609s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665044296 (605h37m44.536127341s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665472091 (605h37m44.536555136s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.665975500 (605h37m44.537058535s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.666446947 (605h37m44.537529992s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.668093252 (605h37m44.539176298s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.669609279 (605h37m44.540692324s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.671474750 (605h37m44.542557785s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.673639683 (605h37m44.544722728s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675503993 (605h37m44.546587028s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.675984438 (605h37m44.547067473s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678017409 (605h37m44.549100474s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.678630597 (605h37m44.549713632s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680256863 (605h37m44.551339898s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:44.680426366 (605h37m44.551509401s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Uploading piece... Dec 30 21:06:46.378086153 (605h37m46.249169188s) [upload=530800]: Next returned nil segment Dec 30 21:06:46.378098546 (605h37m46.249181581s) [upload=530800]: Waiting for error group managing segments... Dec 30 21:06:46.689509938 (605h37m46.560592983s) [noise=true , address=145.79.227.10:4402 , piece_id=LVSJIATUOF62JFWE2D53J5YE6GSYXDGZL5ZRMBRHKI4PJDASRAWA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.690435611 (605h37m46.561518656s) [noise=true , address=145.79.227.41:4085 , piece_id=UGB6OPHBIYWVSNHY55QPTZK4RMA25LHVVXZH4YIBDUN3SPTZT3BQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.692454876 (605h37m46.563537911s) [noise=true , address=145.79.227.47:4267 , piece_id=GAL6HUJZCXCJNV22KLBYE4GZRUIICTDWEXZ3IG2E2D3YUEGM346A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.693387142 (605h37m46.564470187s) [noise=true , address=145.79.227.36:4519 , piece_id=QL6LHYUOJAAFLYGJTPLGFLZZJTJ6Y5HWWABEEDUGE3RHM7HM665A, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.732012909 (605h37m46.603095944s) [noise=true , address=145.79.227.27:4691 , piece_id=UITQV7R23FZWGLZ5ET2BKTTJIJHB66RZZ3BPWO3J5LK4G5BFKVXA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.734087900 (605h37m46.605170945s) [noise=true , address=145.79.227.2:4114 , piece_id=PHDVDUSGHNN2AIDRPGCUJS2IUEKIIVUS56R5R5RX24UADOPAPMNA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.744554940 (605h37m46.615637985s) [noise=true , address=145.79.227.21:4474 , piece_id=3W3CAZDA4BSCRYN2YIAAVLUUXODZTAJ3RP2VZVOLTP6YRY43VB7Q, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.842263910 (605h37m46.713346955s) [noise=true , address=77.90.31.18:4808 , piece_id=ZBYI6RQW3RIVCOUSLWLFJ6TT7CX76GUMX2UW2I6P7HTFIPAIA2IQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.868062052 (605h37m46.739145087s) [noise=true , address=77.90.31.14:4663 , piece_id=5VOZLQWRXWS5I6CNKBOFRUSV4R2G47DNMJ3QDNL7ROT5DICLOZCQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.957435419 (605h37m46.828518454s) [noise=true , address=45.91.116.13:4154 , piece_id=ZWPUE3A5DOPPIN7K2XI5PDSDWWQFUWEHVZWAUDE4Y2IXEZUWOHYQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:46.993198967 (605h37m46.864282012s) [noise=true , address=45.91.116.9:4263 , piece_id=SMSSDMKUFIN3WLBDP5IHFXG7URGERVNBY3ACJRX45QJUKOPP66DA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.234223986 (605h37m47.105307021s) [noise=true , address=77.90.31.19:4842 , piece_id=XE2JXUO25EAVPGSAHJ3INJ4C63XZSYGZOP3NTGOJAZOP7YUL3SKQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686929288 (605h37m47.558012323s) [noise=true , address=45.91.116.8:4045 , piece_id=MNQTLTFEYRKNEV4LRDEFS5KTLKPZTH6YB7OA3JCPNFMUHC2Q2LJA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: <nil> Dec 30 21:06:47.686948274 (605h37m47.558031309s) [seg_pos={0 0}, upload=530800]: Segment reached optimal threshold of 13 pieces. Dec 30 21:06:47.687235953 (605h37m47.558318998s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687243797 (605h37m47.558326832s) [noise=true , address=145.79.227.43:4158 , piece_id=XSY5KFZR5MTHATJVMX2IHB75R6IEP2USCRXRTMWWKQUCZ6VY2ZGA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687275458 (605h37m47.558358513s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687287991 (605h37m47.558371026s) [noise=true , address=145.79.227.40:4051 , piece_id=CL3VTGZTRVRTIT5JFKANBNYPAUBYMY7N6X72APBQYMABFHDLWMQQ, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687497340 (605h37m47.558580385s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 30 21:06:47.687506397 (605h37m47.558589442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5XP7E44ORCG7GBGBQWH5WK5NKRZL36KQIKC4V4LDLRDG6XUPAZA, seg_pos={0 0}, upload=530800]: Long tail cancellation due to optimal threshold Dec 30 21:06:47.687524051 (605h37m47.558607086s) [seg_pos={0 0}, upload=530800]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:06:47.687610214 (605h37m47.558693249s) [upload=530800]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:06:47.687617168 (605h37m47.558700214s) [upload=530800]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:06:47.992181223 (605h37m47.863264258s) [upload=530800]: Done uploading segments... Dec 30 21:06:47.992199908 (605h37m47.863282953s) [upload=530800]: Upload finished. err: <nil> Duration: 30.004038077s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X......X..............................] duration: 3.560988954s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:07:02 UTC) orbiter warning: context deadline exceeded Duration: 3.561019031s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X.......X.X.....X.......................................X] duration: 30.002561475s trace: https://jaeger.dev.storj.io/trace/510daa5fcce9f517 FAIL (2025-12-30 21:06:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HKKUWLZXLNKKJ5YUYF6M3SHDAN2KOBAG3APIO32SFAI3C2YJNXZQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: HG5IJI74BANODJW3WOK6TGIWKFXIB2LWT23HWKG3LCPT7TKUS33Q) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: IIK7BEWE7MGOTVHNQMRSIPPVC3AWTYFKMZLXO7GP53KD7PM5CS6Q) context deadline exceeded ======LOGS====== 3 (605h36m44.363137098s) [upload=530788]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 21:05:44.739664270 (605h36m44.610747305s) [seg_pos={0 0}, upload=530788]: Begin upload segment... Dec 30 21:05:44.739885351 (605h36m44.610968386s) [seg_pos={0 0}, upload=530788]: Done begin upload segment. Dec 30 21:05:44.739900179 (605h36m44.610983224s) [noise=true , address=45.91.116.13:4154 , piece_id=IIK7BEWE7MGOTVHNQMRSIPPVC3AWTYFKMZLXO7GP53KD7PM5CS6Q, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.739992926 (605h36m44.611075981s) [noise=true , address=77.90.31.21:4878 , piece_id=ZIFDBXXIKZQGM3KHGRNA7AAPKWTPBEXEG6HZ2HTYA7FOGEH4JWIQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.740573892 (605h36m44.611656927s) [noise=true , address=77.90.31.17:4773 , piece_id=7XIKC7PWTQABGBDPUYF2NB2YYI7QWRHTIYHRRNNLRFAUPJG2WTAA, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.740919681 (605h36m44.612002716s) [noise=true , address=45.91.116.10:4081 , piece_id=7KZVEWXEJII6RGDUSAZDFV7ATXSJQBY2SLVPYXVWCCLHHYGN7F6Q, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.741230803 (605h36m44.612313838s) [noise=true , address=145.79.227.48:4303 , piece_id=5ENKXV7MPBY52ZQG7Q76ZAXGMNFWGZNKM3O3CETROGKWRBEH2SMQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.741523821 (605h36m44.612606866s) [noise=true , address=77.90.31.19:4841 , piece_id=HKKUWLZXLNKKJ5YUYF6M3SHDAN2KOBAG3APIO32SFAI3C2YJNXZQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.741868127 (605h36m44.612951162s) [noise=true , address=45.91.116.12:4226 , piece_id=DARKZPO5ZNABA24ARU2MKG4K4BK3BYH5K4R4FSXRLKF5GOV3WLDA, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.742172116 (605h36m44.613255151s) [noise=true , address=45.91.116.9:4261 , piece_id=HG5IJI74BANODJW3WOK6TGIWKFXIB2LWT23HWKG3LCPT7TKUS33Q, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.742501482 (605h36m44.613584517s) [noise=true , address=145.79.227.50:4373 , piece_id=A6N7KRAZYVSIF2APVWEKOQ7CBX5IM4QKK4CSKYV34NAN26DV3EWQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.742861398 (605h36m44.613944473s) [noise=true , address=145.79.227.24:4581 , piece_id=P4MS56U43MASPIJG6U65FIRFJFCLP5VBZYWNNVDVK7J2TWI2L2XQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.743250319 (605h36m44.614333354s) [noise=true , address=145.79.227.45:4231 , piece_id=4IHILPKGUXDPSWMTQR4OGQHN4HEXZMKAKRO2KL3ZN4R5JJ2NVEFQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.743600475 (605h36m44.614683521s) [noise=true , address=145.79.227.27:4691 , piece_id=DSZXO7M5QLC6QKJ2NBQO2TJ7WMMC35FNAKW527QCCUJWKVGQKB2A, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.745046027 (605h36m44.616129063s) [noise=true , address=145.79.227.11:4732 , piece_id=7XIO6TTXCQ6FWPAR6GBM4XIK2RXGJT4JB52KD7ABPIFGNB2UMXYA, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.746707091 (605h36m44.617790126s) [noise=true , address=145.79.227.6:4257 , piece_id=TNZE3KR47ZIM7ICQYPX6ILKLNRR4HWNADETBRF7ZCAJXOKZWTHRQ, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.748028838 (605h36m44.619111873s) [noise=true , address=145.79.227.15:4873 , piece_id=SGIFF4KZSI2JEFQFAHQGMULWZTAAKBNXTXWG4UGVFC45WAIGFDPA, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:44.749531188 (605h36m44.620614223s) [noise=true , address=77.90.31.12:4589 , piece_id=DNNVHRLOPGSUK4ICWMPW4ZSKXCN3YQR2BHXWDC3SYOLPZSEYC4LA, seg_pos={0 0}, upload=530788]: Uploading piece... Dec 30 21:05:46.449117764 (605h36m46.320200799s) [upload=530788]: Next returned nil segment Dec 30 21:05:46.449129065 (605h36m46.32021211s) [upload=530788]: Waiting for error group managing segments... Dec 30 21:05:46.728411661 (605h36m46.599494706s) [noise=true , address=145.79.227.45:4231 , piece_id=4IHILPKGUXDPSWMTQR4OGQHN4HEXZMKAKRO2KL3ZN4R5JJ2NVEFQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.740660814 (605h36m46.611743849s) [noise=true , address=145.79.227.24:4581 , piece_id=P4MS56U43MASPIJG6U65FIRFJFCLP5VBZYWNNVDVK7J2TWI2L2XQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.743014717 (605h36m46.614097752s) [noise=true , address=145.79.227.50:4373 , piece_id=A6N7KRAZYVSIF2APVWEKOQ7CBX5IM4QKK4CSKYV34NAN26DV3EWQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.747267274 (605h36m46.618350319s) [noise=true , address=145.79.227.27:4691 , piece_id=DSZXO7M5QLC6QKJ2NBQO2TJ7WMMC35FNAKW527QCCUJWKVGQKB2A, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.748193267 (605h36m46.619276302s) [noise=true , address=145.79.227.6:4257 , piece_id=TNZE3KR47ZIM7ICQYPX6ILKLNRR4HWNADETBRF7ZCAJXOKZWTHRQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.754320144 (605h36m46.625403189s) [noise=true , address=145.79.227.15:4873 , piece_id=SGIFF4KZSI2JEFQFAHQGMULWZTAAKBNXTXWG4UGVFC45WAIGFDPA, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.804449996 (605h36m46.675533041s) [noise=true , address=145.79.227.11:4732 , piece_id=7XIO6TTXCQ6FWPAR6GBM4XIK2RXGJT4JB52KD7ABPIFGNB2UMXYA, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:46.922911010 (605h36m46.793994055s) [noise=true , address=77.90.31.19:4841 , piece_id=HKKUWLZXLNKKJ5YUYF6M3SHDAN2KOBAG3APIO32SFAI3C2YJNXZQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:47.001592509 (605h36m46.872675564s) [noise=true , address=77.90.31.21:4878 , piece_id=ZIFDBXXIKZQGM3KHGRNA7AAPKWTPBEXEG6HZ2HTYA7FOGEH4JWIQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:47.031028438 (605h36m46.902111483s) [noise=true , address=77.90.31.17:4773 , piece_id=7XIKC7PWTQABGBDPUYF2NB2YYI7QWRHTIYHRRNNLRFAUPJG2WTAA, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:47.043056140 (605h36m46.914139175s) [noise=true , address=45.91.116.9:4261 , piece_id=HG5IJI74BANODJW3WOK6TGIWKFXIB2LWT23HWKG3LCPT7TKUS33Q, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:47.053496299 (605h36m46.924579344s) [noise=true , address=45.91.116.13:4154 , piece_id=IIK7BEWE7MGOTVHNQMRSIPPVC3AWTYFKMZLXO7GP53KD7PM5CS6Q, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:47.093741912 (605h36m46.964824947s) [noise=true , address=45.91.116.10:4081 , piece_id=7KZVEWXEJII6RGDUSAZDFV7ATXSJQBY2SLVPYXVWCCLHHYGN7F6Q, seg_pos={0 0}, upload=530788]: Done uploading piece. err: <nil> Dec 30 21:05:47.093766257 (605h36m46.964849292s) [seg_pos={0 0}, upload=530788]: Segment reached optimal threshold of 13 pieces. Dec 30 21:05:47.094207498 (605h36m46.965290533s) [noise=true , address=145.79.227.48:4303 , piece_id=5ENKXV7MPBY52ZQG7Q76ZAXGMNFWGZNKM3O3CETROGKWRBEH2SMQ, seg_pos={0 0}, upload=530788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:45124->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:45124->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45124->145.79.227.48:4303: use of closed network connection Dec 30 21:05:47.094226003 (605h36m46.965309038s) [noise=true , address=145.79.227.48:4303 , piece_id=5ENKXV7MPBY52ZQG7Q76ZAXGMNFWGZNKM3O3CETROGKWRBEH2SMQ, seg_pos={0 0}, upload=530788]: Long tail cancellation due to optimal threshold Dec 30 21:05:47.094250109 (605h36m46.965333144s) [noise=true , address=77.90.31.12:4589 , piece_id=DNNVHRLOPGSUK4ICWMPW4ZSKXCN3YQR2BHXWDC3SYOLPZSEYC4LA, seg_pos={0 0}, upload=530788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:58442->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:58442->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58442->77.90.31.12:4589: use of closed network connection Dec 30 21:05:47.094259326 (605h36m46.965342361s) [noise=true , address=77.90.31.12:4589 , piece_id=DNNVHRLOPGSUK4ICWMPW4ZSKXCN3YQR2BHXWDC3SYOLPZSEYC4LA, seg_pos={0 0}, upload=530788]: Long tail cancellation due to optimal threshold Dec 30 21:05:47.094388823 (605h36m46.965471858s) [noise=true , address=45.91.116.12:4226 , piece_id=DARKZPO5ZNABA24ARU2MKG4K4BK3BYH5K4R4FSXRLKF5GOV3WLDA, seg_pos={0 0}, upload=530788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:48130->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:48130->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48130->45.91.116.12:4226: use of closed network connection Dec 30 21:05:47.094395986 (605h36m46.965479021s) [noise=true , address=45.91.116.12:4226 , piece_id=DARKZPO5ZNABA24ARU2MKG4K4BK3BYH5K4R4FSXRLKF5GOV3WLDA, seg_pos={0 0}, upload=530788]: Long tail cancellation due to optimal threshold Dec 30 21:05:47.094416686 (605h36m46.965499721s) [seg_pos={0 0}, upload=530788]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 21:05:47.094503621 (605h36m46.965586656s) [upload=530788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 21:05:47.094509794 (605h36m46.965592839s) [upload=530788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 21:05:47.387377637 (605h36m47.258460672s) [upload=530788]: Done uploading segments... Dec 30 21:05:47.387397244 (605h36m47.25848028s) [upload=530788]: Upload finished. err: <nil> Duration:* 30.002583487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X......X.............................] duration: 3.565078605s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:06:12 UTC) orbiter warning: context deadline exceeded Duration: 3.565104774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X......X............................] duration: 3.511184657s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:05:22 UTC) orbiter warning: context deadline exceeded Duration: 3.511273206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X......X...........................] duration: 3.603731472s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:04:54 UTC) orbiter warning: context deadline exceeded Duration: 3.603769314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X.......X.X.....X.......................................] duration: 6.666395964s trace: https://jaeger.dev.storj.io/trace/5ee57765719af9d7 FAIL (2025-12-30 21:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 20:35:36.462484063 (605h6m36.333567098s) [upload=530350]: Starting upload Dec 30 20:35:36.462541563 (605h6m36.333624598s) [upload=530350]: Done starting upload Dec 30 20:35:36.462631473 (605h6m36.333714508s) [upload=530350]: Uploading segments... Dec 30 20:35:36.462732586 (605h6m36.333815621s) [upload=530350]: Got next segment. Inline: false Dec 30 20:35:36.462774606 (605h6m36.333857651s) [upload=530350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 20:35:36.462779455 (605h6m36.3338625s) [upload=530350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 20:35:36.708373053 (605h6m36.579456098s) [seg_pos={0 0}, upload=530350]: Begin upload segment... Dec 30 20:35:36.708418520 (605h6m36.579501565s) [seg_pos={0 0}, upload=530350]: Done begin upload segment. Dec 30 20:35:36.708441073 (605h6m36.579524118s) [noise=true , address=145.79.227.22:4510 , piece_id=2W3MPG7RAXR2SI263O7DT6Z4HBH5CBWISJGXTYD75RVA2SMZUYGA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.708453075 (605h6m36.57953613s) [noise=true , address=145.79.227.32:4836 , piece_id=SNR7OR6U3RXAIXFXVGPX7PRUHPHMC6NLPYNYREG757HIZOYGFMWA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.709931441 (605h6m36.581014476s) [noise=true , address=145.79.227.2:4113 , piece_id=Z73OX4M5NF6GMS7UWIKERITX47EYIWKUA75GVWRIYP6M7NSX3N5A, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.710435350 (605h6m36.581518425s) [noise=true , address=77.90.31.10:4519 , piece_id=3OTXFODC7RWYKXZ5KZDJOYFK5R4ROBXLYX7VGQCXDANBNZZEPGNA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.710863575 (605h6m36.58194661s) [noise=true , address=145.79.227.45:4229 , piece_id=7Y6CSE24TKVMEW4QYVAKVUHMUJ5RI7DEDJL3XDN2XVBA5C4CU6QQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.711173525 (605h6m36.58225656s) [noise=true , address=45.91.116.6:4975 , piece_id=VJ7RRLBBNJ3MNXJHVRZ5PGZZTLEFALWMX5TMM2BYJSQERGA5VNCA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.711567595 (605h6m36.582650631s) [noise=true , address=77.90.31.19:4844 , piece_id=MS7CKJW7OSSDQFPLWGGKTXLQA4HAZBLGHBEQHASETHFB5ASGRHQQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.711888587 (605h6m36.582971623s) [noise=true , address=45.91.116.9:4261 , piece_id=S6S63EYX75O43J4NOCLJJVOBP6AASL5TWATSETMQKMWJTYFFSB5Q, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.712302655 (605h6m36.5833857s) [noise=true , address=145.79.227.23:4547 , piece_id=WNZYRVKOSCXRCIKPVRWMKFFPI3HHWNATV7SYPJII2ZVJFFSAZXJQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.712755567 (605h6m36.583838603s) [noise=true , address=77.90.31.11:4556 , piece_id=AYFHYVZZNLSPPZEMXGA3TDRRNB66KUKFLGO6JJA63AZG2C4T4REA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.713196287 (605h6m36.584279322s) [noise=true , address=77.90.31.15:4697 , piece_id=FGF5V4UDVPL4CN76WX2DH5U5XVGJHHQSPAWZRPIXOQWQJHUCPAEQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.713608953 (605h6m36.584691988s) [noise=true , address=45.91.116.7:4012 , piece_id=NLRRC4REW2J7YHJMNTSKDGFZNXMMDLGESCYIECVWWZNWFK7TTNAA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.713891290 (605h6m36.584974325s) [noise=true , address=45.91.116.5:4300 , piece_id=7VUARNDNLPJO7VAU62BOP7AHG4HEP7ZOIR7YUTUJSDMMFKTSCGHA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.714249372 (605h6m36.585332417s) [noise=true , address=145.79.227.25:4619 , piece_id=UWBDUI3ZAFHH333RZ7EPRTLIY4F4LUROTMZM3UOTQFRCW7K55HTA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.714637331 (605h6m36.585720366s) [noise=true , address=145.79.227.47:4267 , piece_id=LMDMZG33BFWBFSMHPYEQM3JRHZANEMN7RKGGCOJ3MIV5TGHIHEFQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.715012355 (605h6m36.58609539s) [noise=true , address=145.79.227.14:4837 , piece_id=PIKGAVN4VRUF3C3UYLMUXTBCJJTCAOOCPTOLFBVIREJJK5RGVNZA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:38.316292430 (605h6m38.187375465s) [upload=530350]: Next returned nil segment Dec 30 20:35:38.316341784 (605h6m38.187424819s) [upload=530350]: Waiting for error group managing segments... Dec 30 20:35:38.701193221 (605h6m38.572276266s) [noise=true , address=145.79.227.23:4547 , piece_id=WNZYRVKOSCXRCIKPVRWMKFFPI3HHWNATV7SYPJII2ZVJFFSAZXJQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.705327784 (605h6m38.576410819s) [noise=true , address=145.79.227.32:4836 , piece_id=SNR7OR6U3RXAIXFXVGPX7PRUHPHMC6NLPYNYREG757HIZOYGFMWA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.712186354 (605h6m38.583269399s) [noise=true , address=145.79.227.47:4267 , piece_id=LMDMZG33BFWBFSMHPYEQM3JRHZANEMN7RKGGCOJ3MIV5TGHIHEFQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.716484538 (605h6m38.587567583s) [noise=true , address=145.79.227.2:4113 , piece_id=Z73OX4M5NF6GMS7UWIKERITX47EYIWKUA75GVWRIYP6M7NSX3N5A, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.721546737 (605h6m38.592629782s) [noise=true , address=145.79.227.22:4510 , piece_id=2W3MPG7RAXR2SI263O7DT6Z4HBH5CBWISJGXTYD75RVA2SMZUYGA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.770544116 (605h6m38.641627151s) [noise=true , address=145.79.227.14:4837 , piece_id=PIKGAVN4VRUF3C3UYLMUXTBCJJTCAOOCPTOLFBVIREJJK5RGVNZA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.782075523 (605h6m38.653158558s) [noise=true , address=145.79.227.45:4229 , piece_id=7Y6CSE24TKVMEW4QYVAKVUHMUJ5RI7DEDJL3XDN2XVBA5C4CU6QQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.868556056 (605h6m38.739639091s) [noise=true , address=145.79.227.25:4619 , piece_id=UWBDUI3ZAFHH333RZ7EPRTLIY4F4LUROTMZM3UOTQFRCW7K55HTA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.868598947 (605h6m38.739681992s) [noise=true , address=77.90.31.15:4697 , piece_id=FGF5V4UDVPL4CN76WX2DH5U5XVGJHHQSPAWZRPIXOQWQJHUCPAEQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.901249089 (605h6m38.772332124s) [noise=true , address=77.90.31.10:4519 , piece_id=3OTXFODC7RWYKXZ5KZDJOYFK5R4ROBXLYX7VGQCXDANBNZZEPGNA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.004123206 (605h6m38.875206282s) [noise=true , address=45.91.116.7:4012 , piece_id=NLRRC4REW2J7YHJMNTSKDGFZNXMMDLGESCYIECVWWZNWFK7TTNAA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.014034158 (605h6m38.885117223s) [noise=true , address=77.90.31.11:4556 , piece_id=AYFHYVZZNLSPPZEMXGA3TDRRNB66KUKFLGO6JJA63AZG2C4T4REA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.015040745 (605h6m38.88612378s) [noise=true , address=45.91.116.6:4975 , piece_id=VJ7RRLBBNJ3MNXJHVRZ5PGZZTLEFALWMX5TMM2BYJSQERGA5VNCA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.015061114 (605h6m38.886144149s) [seg_pos={0 0}, upload=530350]: Segment reached optimal threshold of 13 pieces. Dec 30 20:35:39.015395129 (605h6m38.886478164s) [noise=true , address=45.91.116.5:4300 , piece_id=7VUARNDNLPJO7VAU62BOP7AHG4HEP7ZOIR7YUTUJSDMMFKTSCGHA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 20:35:39.015411591 (605h6m38.886494626s) [noise=true , address=45.91.116.5:4300 , piece_id=7VUARNDNLPJO7VAU62BOP7AHG4HEP7ZOIR7YUTUJSDMMFKTSCGHA, seg_pos={0 0}, upload=530350]: Long tail cancellation due to optimal threshold Dec 30 20:35:39.015743864 (605h6m38.886826909s) [noise=true , address=77.90.31.19:4844 , piece_id=MS7CKJW7OSSDQFPLWGGKTXLQA4HAZBLGHBEQHASETHFB5ASGRHQQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 20:35:39.015761427 (605h6m38.886844472s) [noise=true , address=77.90.31.19:4844 , piece_id=MS7CKJW7OSSDQFPLWGGKTXLQA4HAZBLGHBEQHASETHFB5ASGRHQQ, seg_pos={0 0}, upload=530350]: Long tail cancellation due to optimal threshold Dec 30 20:35:39.015794691 (605h6m38.886877736s) [noise=true , address=45.91.116.9:4261 , piece_id=S6S63EYX75O43J4NOCLJJVOBP6AASL5TWATSETMQKMWJTYFFSB5Q, seg_pos={0 0}, upload=530350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:33964->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 20:35:39.015805060 (605h6m38.886888105s) [noise=true , address=45.91.116.9:4261 , piece_id=S6S63EYX75O43J4NOCLJJVOBP6AASL5TWATSETMQKMWJTYFFSB5Q, seg_pos={0 0}, upload=530350]: Long tail cancellation due to optimal threshold Dec 30 20:35:39.015826191 (605h6m38.886909236s) [seg_pos={0 0}, upload=530350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 20:35:39.016045879 (605h6m38.887128924s) [upload=530350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 20:35:39.016089141 (605h6m38.887172216s) [upload=530350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 20:35:39.310568526 (605h6m39.181651561s) [upload=530350]: Done uploading segments... Dec 30 20:35:39.310586500 (605h6m39.181669535s) [upload=530350]: Upload finished. err: <nil> Duration: 6.666439899s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X......X..........................] duration: 3.583234083s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:04:15 UTC) orbiter warning: context deadline exceeded Duration: 3.583260112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X......X.........................] duration: 3.567774346s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:03:52 UTC) orbiter warning: context deadline exceeded Duration: 3.567799113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X.......X.X.....X......................................] duration: 6.166421435s trace: https://jaeger.dev.storj.io/trace/5ee57765719af9d7 FAIL (2025-12-30 21:03:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 20:35:36.462484063 (605h6m36.333567098s) [upload=530350]: Starting upload Dec 30 20:35:36.462541563 (605h6m36.333624598s) [upload=530350]: Done starting upload Dec 30 20:35:36.462631473 (605h6m36.333714508s) [upload=530350]: Uploading segments... Dec 30 20:35:36.462732586 (605h6m36.333815621s) [upload=530350]: Got next segment. Inline: false Dec 30 20:35:36.462774606 (605h6m36.333857651s) [upload=530350]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 20:35:36.462779455 (605h6m36.3338625s) [upload=530350]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 20:35:36.708373053 (605h6m36.579456098s) [seg_pos={0 0}, upload=530350]: Begin upload segment... Dec 30 20:35:36.708418520 (605h6m36.579501565s) [seg_pos={0 0}, upload=530350]: Done begin upload segment. Dec 30 20:35:36.708441073 (605h6m36.579524118s) [noise=true , address=145.79.227.22:4510 , piece_id=2W3MPG7RAXR2SI263O7DT6Z4HBH5CBWISJGXTYD75RVA2SMZUYGA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.708453075 (605h6m36.57953613s) [noise=true , address=145.79.227.32:4836 , piece_id=SNR7OR6U3RXAIXFXVGPX7PRUHPHMC6NLPYNYREG757HIZOYGFMWA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.709931441 (605h6m36.581014476s) [noise=true , address=145.79.227.2:4113 , piece_id=Z73OX4M5NF6GMS7UWIKERITX47EYIWKUA75GVWRIYP6M7NSX3N5A, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.710435350 (605h6m36.581518425s) [noise=true , address=77.90.31.10:4519 , piece_id=3OTXFODC7RWYKXZ5KZDJOYFK5R4ROBXLYX7VGQCXDANBNZZEPGNA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.710863575 (605h6m36.58194661s) [noise=true , address=145.79.227.45:4229 , piece_id=7Y6CSE24TKVMEW4QYVAKVUHMUJ5RI7DEDJL3XDN2XVBA5C4CU6QQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.711173525 (605h6m36.58225656s) [noise=true , address=45.91.116.6:4975 , piece_id=VJ7RRLBBNJ3MNXJHVRZ5PGZZTLEFALWMX5TMM2BYJSQERGA5VNCA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.711567595 (605h6m36.582650631s) [noise=true , address=77.90.31.19:4844 , piece_id=MS7CKJW7OSSDQFPLWGGKTXLQA4HAZBLGHBEQHASETHFB5ASGRHQQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.711888587 (605h6m36.582971623s) [noise=true , address=45.91.116.9:4261 , piece_id=S6S63EYX75O43J4NOCLJJVOBP6AASL5TWATSETMQKMWJTYFFSB5Q, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.712302655 (605h6m36.5833857s) [noise=true , address=145.79.227.23:4547 , piece_id=WNZYRVKOSCXRCIKPVRWMKFFPI3HHWNATV7SYPJII2ZVJFFSAZXJQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.712755567 (605h6m36.583838603s) [noise=true , address=77.90.31.11:4556 , piece_id=AYFHYVZZNLSPPZEMXGA3TDRRNB66KUKFLGO6JJA63AZG2C4T4REA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.713196287 (605h6m36.584279322s) [noise=true , address=77.90.31.15:4697 , piece_id=FGF5V4UDVPL4CN76WX2DH5U5XVGJHHQSPAWZRPIXOQWQJHUCPAEQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.713608953 (605h6m36.584691988s) [noise=true , address=45.91.116.7:4012 , piece_id=NLRRC4REW2J7YHJMNTSKDGFZNXMMDLGESCYIECVWWZNWFK7TTNAA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.713891290 (605h6m36.584974325s) [noise=true , address=45.91.116.5:4300 , piece_id=7VUARNDNLPJO7VAU62BOP7AHG4HEP7ZOIR7YUTUJSDMMFKTSCGHA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.714249372 (605h6m36.585332417s) [noise=true , address=145.79.227.25:4619 , piece_id=UWBDUI3ZAFHH333RZ7EPRTLIY4F4LUROTMZM3UOTQFRCW7K55HTA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.714637331 (605h6m36.585720366s) [noise=true , address=145.79.227.47:4267 , piece_id=LMDMZG33BFWBFSMHPYEQM3JRHZANEMN7RKGGCOJ3MIV5TGHIHEFQ, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:36.715012355 (605h6m36.58609539s) [noise=true , address=145.79.227.14:4837 , piece_id=PIKGAVN4VRUF3C3UYLMUXTBCJJTCAOOCPTOLFBVIREJJK5RGVNZA, seg_pos={0 0}, upload=530350]: Uploading piece... Dec 30 20:35:38.316292430 (605h6m38.187375465s) [upload=530350]: Next returned nil segment Dec 30 20:35:38.316341784 (605h6m38.187424819s) [upload=530350]: Waiting for error group managing segments... Dec 30 20:35:38.701193221 (605h6m38.572276266s) [noise=true , address=145.79.227.23:4547 , piece_id=WNZYRVKOSCXRCIKPVRWMKFFPI3HHWNATV7SYPJII2ZVJFFSAZXJQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.705327784 (605h6m38.576410819s) [noise=true , address=145.79.227.32:4836 , piece_id=SNR7OR6U3RXAIXFXVGPX7PRUHPHMC6NLPYNYREG757HIZOYGFMWA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.712186354 (605h6m38.583269399s) [noise=true , address=145.79.227.47:4267 , piece_id=LMDMZG33BFWBFSMHPYEQM3JRHZANEMN7RKGGCOJ3MIV5TGHIHEFQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.716484538 (605h6m38.587567583s) [noise=true , address=145.79.227.2:4113 , piece_id=Z73OX4M5NF6GMS7UWIKERITX47EYIWKUA75GVWRIYP6M7NSX3N5A, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.721546737 (605h6m38.592629782s) [noise=true , address=145.79.227.22:4510 , piece_id=2W3MPG7RAXR2SI263O7DT6Z4HBH5CBWISJGXTYD75RVA2SMZUYGA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.770544116 (605h6m38.641627151s) [noise=true , address=145.79.227.14:4837 , piece_id=PIKGAVN4VRUF3C3UYLMUXTBCJJTCAOOCPTOLFBVIREJJK5RGVNZA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.782075523 (605h6m38.653158558s) [noise=true , address=145.79.227.45:4229 , piece_id=7Y6CSE24TKVMEW4QYVAKVUHMUJ5RI7DEDJL3XDN2XVBA5C4CU6QQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.868556056 (605h6m38.739639091s) [noise=true , address=145.79.227.25:4619 , piece_id=UWBDUI3ZAFHH333RZ7EPRTLIY4F4LUROTMZM3UOTQFRCW7K55HTA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.868598947 (605h6m38.739681992s) [noise=true , address=77.90.31.15:4697 , piece_id=FGF5V4UDVPL4CN76WX2DH5U5XVGJHHQSPAWZRPIXOQWQJHUCPAEQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:38.901249089 (605h6m38.772332124s) [noise=true , address=77.90.31.10:4519 , piece_id=3OTXFODC7RWYKXZ5KZDJOYFK5R4ROBXLYX7VGQCXDANBNZZEPGNA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.004123206 (605h6m38.875206282s) [noise=true , address=45.91.116.7:4012 , piece_id=NLRRC4REW2J7YHJMNTSKDGFZNXMMDLGESCYIECVWWZNWFK7TTNAA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.014034158 (605h6m38.885117223s) [noise=true , address=77.90.31.11:4556 , piece_id=AYFHYVZZNLSPPZEMXGA3TDRRNB66KUKFLGO6JJA63AZG2C4T4REA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.015040745 (605h6m38.88612378s) [noise=true , address=45.91.116.6:4975 , piece_id=VJ7RRLBBNJ3MNXJHVRZ5PGZZTLEFALWMX5TMM2BYJSQERGA5VNCA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: <nil> Dec 30 20:35:39.015061114 (605h6m38.886144149s) [seg_pos={0 0}, upload=530350]: Segment reached optimal threshold of 13 pieces. Dec 30 20:35:39.015395129 (605h6m38.886478164s) [noise=true , address=45.91.116.5:4300 , piece_id=7VUARNDNLPJO7VAU62BOP7AHG4HEP7ZOIR7YUTUJSDMMFKTSCGHA, seg_pos={0 0}, upload=530350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 20:35:39.015411591 (605h6m38.886494626s) [noise=true , address=45.91.116.5:4300 , piece_id=7VUARNDNLPJO7VAU62BOP7AHG4HEP7ZOIR7YUTUJSDMMFKTSCGHA, seg_pos={0 0}, upload=530350]: Long tail cancellation due to optimal threshold Dec 30 20:35:39.015743864 (605h6m38.886826909s) [noise=true , address=77.90.31.19:4844 , piece_id=MS7CKJW7OSSDQFPLWGGKTXLQA4HAZBLGHBEQHASETHFB5ASGRHQQ, seg_pos={0 0}, upload=530350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 30 20:35:39.015761427 (605h6m38.886844472s) [noise=true , address=77.90.31.19:4844 , piece_id=MS7CKJW7OSSDQFPLWGGKTXLQA4HAZBLGHBEQHASETHFB5ASGRHQQ, seg_pos={0 0}, upload=530350]: Long tail cancellation due to optimal threshold Dec 30 20:35:39.015794691 (605h6m38.886877736s) [noise=true , address=45.91.116.9:4261 , piece_id=S6S63EYX75O43J4NOCLJJVOBP6AASL5TWATSETMQKMWJTYFFSB5Q, seg_pos={0 0}, upload=530350]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: read tcp 188.166.225.170:33964->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 20:35:39.015805060 (605h6m38.886888105s) [noise=true , address=45.91.116.9:4261 , piece_id=S6S63EYX75O43J4NOCLJJVOBP6AASL5TWATSETMQKMWJTYFFSB5Q, seg_pos={0 0}, upload=530350]: Long tail cancellation due to optimal threshold Dec 30 20:35:39.015826191 (605h6m38.886909236s) [seg_pos={0 0}, upload=530350]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 20:35:39.016045879 (605h6m38.887128924s) [upload=530350]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 20:35:39.016089141 (605h6m38.887172216s) [upload=530350]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 20:35:39.310568526 (605h6m39.181651561s) [upload=530350]: Done uploading segments... Dec 30 20:35:39.310586500 (605h6m39.181669535s) [upload=530350]: Upload finished. err: <nil> Duration: 6.166447203s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X......X........................] duration: 3.609068797s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:03:24 UTC) orbiter warning: context deadline exceeded Duration: 3.609164207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X......X.......................] duration: 3.57346391s trace: https://jaeger.dev.storj.io/trace/203a16f4d8059688 FAIL (2025-12-30 21:03:07 UTC) orbiter warning: context deadline exceeded Duration: 3.573486895s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)