Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X..X.................................X...X...] duration: 7.088528298s trace: https://jaeger.dev.storj.io/trace/24e295cb6499d617 FAIL (2026-01-11 15:18:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: 7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ) context deadline exceeded; piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ) context deadline exceeded ======LOGS====== Jan 11 15:15:58.251318463 (887h46m58.122401508s) [upload=776889]: Starting upload Jan 11 15:15:58.251368879 (887h46m58.122451914s) [upload=776889]: Done starting upload Jan 11 15:15:58.251432850 (887h46m58.122515895s) [upload=776889]: Uploading segments... Jan 11 15:15:58.251700771 (887h46m58.122783816s) [upload=776889]: Got next segment. Inline: false Jan 11 15:15:58.251747870 (887h46m58.122830905s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:15:58.251752650 (887h46m58.122835695s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:15:58.496730317 (887h46m58.367813372s) [seg_pos={0 0}, upload=776889]: Begin upload segment... Jan 11 15:15:58.496798697 (887h46m58.367881742s) [seg_pos={0 0}, upload=776889]: Done begin upload segment. Jan 11 15:15:58.496827963 (887h46m58.367910998s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.496835768 (887h46m58.367918803s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497675567 (887h46m58.368758612s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497920874 (887h46m58.369003909s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498291049 (887h46m58.369374084s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498589026 (887h46m58.369672071s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498904246 (887h46m58.369987281s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499297305 (887h46m58.37038034s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499621853 (887h46m58.370704888s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499990385 (887h46m58.37107343s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500347325 (887h46m58.37143036s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500710285 (887h46m58.37179332s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.501069700 (887h46m58.372152736s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502381578 (887h46m58.373464623s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502903321 (887h46m58.373986356s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.503455032 (887h46m58.374538067s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:16:00.187753816 (887h47m0.058836851s) [upload=776889]: Next returned nil segment Jan 11 15:16:00.187767894 (887h47m0.058850929s) [upload=776889]: Waiting for error group managing segments... Jan 11 15:16:00.467254270 (887h47m0.338337305s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.483536243 (887h47m0.354619278s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.485126101 (887h47m0.356209146s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.489280722 (887h47m0.360363767s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.502546541 (887h47m0.373629586s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.505349379 (887h47m0.376432414s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.506378979 (887h47m0.377462014s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.511126849 (887h47m0.382209904s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.512992622 (887h47m0.384075657s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.519559416 (887h47m0.390642461s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.559236496 (887h47m0.430319531s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.564418554 (887h47m0.435501589s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568205815 (887h47m0.43928885s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568231995 (887h47m0.43931503s) [seg_pos={0 0}, upload=776889]: Segment reached optimal threshold of 13 pieces. Jan 11 15:16:00.568478584 (887h47m0.439561619s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568489675 (887h47m0.43957272s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568603171 (887h47m0.439686206s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568758838 (887h47m0.439841893s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568818371 (887h47m0.439901406s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568826176 (887h47m0.439909211s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568847506 (887h47m0.439930541s) [seg_pos={0 0}, upload=776889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:16:00.568920606 (887h47m0.440003651s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:16:00.568925165 (887h47m0.44000821s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:16:00.857315054 (887h47m0.728398089s) [upload=776889]: Done uploading segments... Jan 11 15:16:00.857334662 (887h47m0.728417697s) [upload=776889]: Upload finished. err: <nil> Duration: 7.08855545s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.7461621s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:18:35 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.746187448s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.XXXXXX] duration: 5.000689776s trace: https://jaeger.dev.storj.io/trace/10fcca019c4c0a57 FAIL (2026-01-11 15:18:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000769477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXXXXXXXX.X.XX.XX...X..X] duration: 30.000944498s trace: https://jaeger.dev.storj.io/trace/8929e33c7e6ccc FAIL (2026-01-11 15:18:23 UTC) orbiter warning: context deadline exceeded Duration: 30.000969244s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.969053881s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:18:20 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.969079809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.XXXXX] duration: 5.001463619s trace: https://jaeger.dev.storj.io/trace/11ecb90615aea987 FAIL (2026-01-11 15:18:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001479238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X..X.................................X...X..] duration: 5.663839896s trace: https://jaeger.dev.storj.io/trace/24e295cb6499d617 FAIL (2026-01-11 15:18:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: 7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ) context deadline exceeded; piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ) context deadline exceeded ======LOGS====== Jan 11 15:15:58.251318463 (887h46m58.122401508s) [upload=776889]: Starting upload Jan 11 15:15:58.251368879 (887h46m58.122451914s) [upload=776889]: Done starting upload Jan 11 15:15:58.251432850 (887h46m58.122515895s) [upload=776889]: Uploading segments... Jan 11 15:15:58.251700771 (887h46m58.122783816s) [upload=776889]: Got next segment. Inline: false Jan 11 15:15:58.251747870 (887h46m58.122830905s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:15:58.251752650 (887h46m58.122835695s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:15:58.496730317 (887h46m58.367813372s) [seg_pos={0 0}, upload=776889]: Begin upload segment... Jan 11 15:15:58.496798697 (887h46m58.367881742s) [seg_pos={0 0}, upload=776889]: Done begin upload segment. Jan 11 15:15:58.496827963 (887h46m58.367910998s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.496835768 (887h46m58.367918803s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497675567 (887h46m58.368758612s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497920874 (887h46m58.369003909s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498291049 (887h46m58.369374084s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498589026 (887h46m58.369672071s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498904246 (887h46m58.369987281s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499297305 (887h46m58.37038034s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499621853 (887h46m58.370704888s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499990385 (887h46m58.37107343s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500347325 (887h46m58.37143036s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500710285 (887h46m58.37179332s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.501069700 (887h46m58.372152736s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502381578 (887h46m58.373464623s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502903321 (887h46m58.373986356s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.503455032 (887h46m58.374538067s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:16:00.187753816 (887h47m0.058836851s) [upload=776889]: Next returned nil segment Jan 11 15:16:00.187767894 (887h47m0.058850929s) [upload=776889]: Waiting for error group managing segments... Jan 11 15:16:00.467254270 (887h47m0.338337305s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.483536243 (887h47m0.354619278s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.485126101 (887h47m0.356209146s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.489280722 (887h47m0.360363767s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.502546541 (887h47m0.373629586s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.505349379 (887h47m0.376432414s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.506378979 (887h47m0.377462014s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.511126849 (887h47m0.382209904s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.512992622 (887h47m0.384075657s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.519559416 (887h47m0.390642461s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.559236496 (887h47m0.430319531s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.564418554 (887h47m0.435501589s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568205815 (887h47m0.43928885s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568231995 (887h47m0.43931503s) [seg_pos={0 0}, upload=776889]: Segment reached optimal threshold of 13 pieces. Jan 11 15:16:00.568478584 (887h47m0.439561619s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568489675 (887h47m0.43957272s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568603171 (887h47m0.439686206s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568758838 (887h47m0.439841893s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568818371 (887h47m0.439901406s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568826176 (887h47m0.439909211s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568847506 (887h47m0.439930541s) [seg_pos={0 0}, upload=776889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:16:00.568920606 (887h47m0.440003651s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:16:00.568925165 (887h47m0.44000821s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:16:00.857315054 (887h47m0.728398089s) [upload=776889]: Done uploading segments... Jan 11 15:16:00.857334662 (887h47m0.728417697s) [upload=776889]: Upload finished. err: <nil> Duration: 5.66386833s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X.XX..X.........XX...X....XX.X..X......X.X..................] duration: 21.057722488s trace: https://jaeger.dev.storj.io/trace/3beb10e6790340c6 FAIL (2026-01-11 15:17:58 UTC) orbiter warning: context deadline exceeded Duration: 21.057741683s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.XXXX] duration: 5.001091301s trace: https://jaeger.dev.storj.io/trace/d2bcb4847f446f7 FAIL (2026-01-11 15:17:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001104397s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.425919225s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:17:56 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.425945546s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.X......................................] duration: 6.083813869s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 6.083840762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [XXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXXXXXXXX.X.XX.XX...X..] duration: 28.79547498s trace: https://jaeger.dev.storj.io/trace/29f06ff3cf95c6f5 FAIL (2026-01-11 15:17:48 UTC) orbiter warning: context deadline exceeded Duration: 28.795495058s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.934626417s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:17:41 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.934656023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.888122761s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:17:37 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.888153439s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.X.....................................] duration: 7.316713038s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:17:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 7.316737015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X..X.................................X...X.] duration: 7.216917878s trace: https://jaeger.dev.storj.io/trace/24e295cb6499d617 FAIL (2026-01-11 15:17:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: 7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ) context deadline exceeded; piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ) context deadline exceeded ======LOGS====== Jan 11 15:15:58.251318463 (887h46m58.122401508s) [upload=776889]: Starting upload Jan 11 15:15:58.251368879 (887h46m58.122451914s) [upload=776889]: Done starting upload Jan 11 15:15:58.251432850 (887h46m58.122515895s) [upload=776889]: Uploading segments... Jan 11 15:15:58.251700771 (887h46m58.122783816s) [upload=776889]: Got next segment. Inline: false Jan 11 15:15:58.251747870 (887h46m58.122830905s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:15:58.251752650 (887h46m58.122835695s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:15:58.496730317 (887h46m58.367813372s) [seg_pos={0 0}, upload=776889]: Begin upload segment... Jan 11 15:15:58.496798697 (887h46m58.367881742s) [seg_pos={0 0}, upload=776889]: Done begin upload segment. Jan 11 15:15:58.496827963 (887h46m58.367910998s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.496835768 (887h46m58.367918803s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497675567 (887h46m58.368758612s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497920874 (887h46m58.369003909s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498291049 (887h46m58.369374084s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498589026 (887h46m58.369672071s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498904246 (887h46m58.369987281s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499297305 (887h46m58.37038034s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499621853 (887h46m58.370704888s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499990385 (887h46m58.37107343s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500347325 (887h46m58.37143036s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500710285 (887h46m58.37179332s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.501069700 (887h46m58.372152736s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502381578 (887h46m58.373464623s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502903321 (887h46m58.373986356s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.503455032 (887h46m58.374538067s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:16:00.187753816 (887h47m0.058836851s) [upload=776889]: Next returned nil segment Jan 11 15:16:00.187767894 (887h47m0.058850929s) [upload=776889]: Waiting for error group managing segments... Jan 11 15:16:00.467254270 (887h47m0.338337305s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.483536243 (887h47m0.354619278s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.485126101 (887h47m0.356209146s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.489280722 (887h47m0.360363767s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.502546541 (887h47m0.373629586s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.505349379 (887h47m0.376432414s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.506378979 (887h47m0.377462014s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.511126849 (887h47m0.382209904s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.512992622 (887h47m0.384075657s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.519559416 (887h47m0.390642461s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.559236496 (887h47m0.430319531s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.564418554 (887h47m0.435501589s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568205815 (887h47m0.43928885s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568231995 (887h47m0.43931503s) [seg_pos={0 0}, upload=776889]: Segment reached optimal threshold of 13 pieces. Jan 11 15:16:00.568478584 (887h47m0.439561619s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568489675 (887h47m0.43957272s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568603171 (887h47m0.439686206s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568758838 (887h47m0.439841893s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568818371 (887h47m0.439901406s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568826176 (887h47m0.439909211s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568847506 (887h47m0.439930541s) [seg_pos={0 0}, upload=776889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:16:00.568920606 (887h47m0.440003651s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:16:00.568925165 (887h47m0.44000821s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:16:00.857315054 (887h47m0.728398089s) [upload=776889]: Done uploading segments... Jan 11 15:16:00.857334662 (887h47m0.728417697s) [upload=776889]: Upload finished. err: <nil> Duration: 7.21694987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 45.527984927s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:17:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 45.528008783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.XXX] duration: 5.001395468s trace: https://jaeger.dev.storj.io/trace/23bc873cec6f1c06 FAIL (2026-01-11 15:17:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001421077s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.X.XX..X.........XX...X....XX.X..X......X.X.................] duration: 25.121985386s trace: https://jaeger.dev.storj.io/trace/3beb10e6790340c6 FAIL (2026-01-11 15:16:55 UTC) orbiter warning: context deadline exceeded Duration: 25.121999252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXXXXXXXX.X.XX.XX...X.] duration: 26.544350694s trace: https://jaeger.dev.storj.io/trace/29f06ff3cf95c6f5 FAIL (2026-01-11 15:16:49 UTC) orbiter warning: context deadline exceeded Duration: 26.544371253s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.010016471s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:16:38 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.010040817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.27802014s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:16:36 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 8.278080636s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..X.XX..X.........XX...X....XX.X..X......X.X................] duration: 46.960987937s trace: https://jaeger.dev.storj.io/trace/3beb10e6790340c6 FAIL (2026-01-11 15:16:30 UTC) orbiter warning: context deadline exceeded Duration: 46.961012934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X..X.................................X...X] duration: 30.002627941s trace: https://jaeger.dev.storj.io/trace/24e295cb6499d617 FAIL (2026-01-11 15:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, Piece ID: 7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ) context deadline exceeded; piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ) context deadline exceeded ======LOGS====== Jan 11 15:15:58.251318463 (887h46m58.122401508s) [upload=776889]: Starting upload Jan 11 15:15:58.251368879 (887h46m58.122451914s) [upload=776889]: Done starting upload Jan 11 15:15:58.251432850 (887h46m58.122515895s) [upload=776889]: Uploading segments... Jan 11 15:15:58.251700771 (887h46m58.122783816s) [upload=776889]: Got next segment. Inline: false Jan 11 15:15:58.251747870 (887h46m58.122830905s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:15:58.251752650 (887h46m58.122835695s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:15:58.496730317 (887h46m58.367813372s) [seg_pos={0 0}, upload=776889]: Begin upload segment... Jan 11 15:15:58.496798697 (887h46m58.367881742s) [seg_pos={0 0}, upload=776889]: Done begin upload segment. Jan 11 15:15:58.496827963 (887h46m58.367910998s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.496835768 (887h46m58.367918803s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497675567 (887h46m58.368758612s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.497920874 (887h46m58.369003909s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498291049 (887h46m58.369374084s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498589026 (887h46m58.369672071s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.498904246 (887h46m58.369987281s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499297305 (887h46m58.37038034s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499621853 (887h46m58.370704888s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.499990385 (887h46m58.37107343s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500347325 (887h46m58.37143036s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.500710285 (887h46m58.37179332s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.501069700 (887h46m58.372152736s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502381578 (887h46m58.373464623s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.502903321 (887h46m58.373986356s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:15:58.503455032 (887h46m58.374538067s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Uploading piece... Jan 11 15:16:00.187753816 (887h47m0.058836851s) [upload=776889]: Next returned nil segment Jan 11 15:16:00.187767894 (887h47m0.058850929s) [upload=776889]: Waiting for error group managing segments... Jan 11 15:16:00.467254270 (887h47m0.338337305s) [noise=true , address=145.79.227.2:4114 , piece_id=WOVJRFMCFVG7QH2CDAUMUDJV2MR2F3MFJINNVOODET6QC2Y6ZF3Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.483536243 (887h47m0.354619278s) [noise=true , address=145.79.227.38:4979 , piece_id=FTKNAOMKZDMHYC7K62BPLKB2MLTXSQ7WYTWLV35CAUCUSM5SMAHA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.485126101 (887h47m0.356209146s) [noise=true , address=145.79.227.44:4195 , piece_id=2PBH5OIDE5KHPOVBZGXYQLYKLC3U523O4OKSCUQ4TKKZG7QFGF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.489280722 (887h47m0.360363767s) [noise=true , address=145.79.227.7:4295 , piece_id=7O5U2KJ3JZT33X25TGZXL7G6GMZFSXLAKUDQ3ST3CE2U3BPZVELQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.502546541 (887h47m0.373629586s) [noise=true , address=145.79.227.37:4567 , piece_id=F2DYOKT5RKXBMOC6QZNZGKUFXY7SS2IU6AS5IPTSGH74EIYNI47Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.505349379 (887h47m0.376432414s) [noise=true , address=145.79.227.10:4402 , piece_id=QGSDS6DRIYBAVWPCXRATKTF64CFC56WV3JZ3WN2AT5CC2ZDQHKAQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.506378979 (887h47m0.377462014s) [noise=true , address=145.79.227.19:4078 , piece_id=ZN6DTQS4W7KAVEREWCDF66DIMEVBFM5MJMLJJR2VPPFSYT7WJPIA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.511126849 (887h47m0.382209904s) [noise=true , address=145.79.227.35:4482 , piece_id=NECSD2KMJ5RRRS7ESJG3HTZZ2GYG3PF6F6UIBA2HDDZEKS7Z3EUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.512992622 (887h47m0.384075657s) [noise=true , address=145.79.227.27:4691 , piece_id=ROVIYOP5PJ3QW3BB4FQENTOIRLAZDG4RXZWLJS2KR65IBO7XGDEQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.519559416 (887h47m0.390642461s) [noise=true , address=145.79.227.3:4149 , piece_id=VICKFQEQEQ6FXKN6MAT5OGQ6OL2LFFISIM4R6EZ7BK73BNAYKPUQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.559236496 (887h47m0.430319531s) [noise=true , address=145.79.227.51:4409 , piece_id=WZLV5MCLI265L2CANME6BVWL3IF6H53FKZSQQP2PT53PECXPPT7Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.564418554 (887h47m0.435501589s) [noise=true , address=145.79.227.40:4051 , piece_id=KFCYEPJNLQVXIRA6A2AHKVD3KAR6WZ6D6I564KJIAM4WRG4LOF4Q, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568205815 (887h47m0.43928885s) [noise=true , address=145.79.227.14:4838 , piece_id=DJJ3OVWGASTRNWGDIOHN3DESQVNJ4OBKCVWCQFDJDMFTUYNK4LYA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: <nil> Jan 11 15:16:00.568231995 (887h47m0.43931503s) [seg_pos={0 0}, upload=776889]: Segment reached optimal threshold of 13 pieces. Jan 11 15:16:00.568478584 (887h47m0.439561619s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568489675 (887h47m0.43957272s) [noise=true , address=145.79.227.34:4907 , piece_id=ZD4K336MX3L4GJV7L2L2NNCPBYJDU3OIIPVUNXV37HV3BDI23KRQ, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568603171 (887h47m0.439686206s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568758838 (887h47m0.439841893s) [noise=true , address=145.79.227.31:4797 , piece_id=MAZEIEPOBQPYSEGBQWHN6O6JDCTEXFK3QMZ2MFCBXO53E3HXAZKA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568818371 (887h47m0.439901406s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: expected piece hash; context canceled; EOF Jan 11 15:16:00.568826176 (887h47m0.439909211s) [noise=true , address=145.79.227.11:4732 , piece_id=RZ75GY5YIFJS24ES3O6N2FUSOXWMZDODXZ3FFHJZHRQVZ3WA6TOA, seg_pos={0 0}, upload=776889]: Long tail cancellation due to optimal threshold Jan 11 15:16:00.568847506 (887h47m0.439930541s) [seg_pos={0 0}, upload=776889]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:16:00.568920606 (887h47m0.440003651s) [upload=776889]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:16:00.568925165 (887h47m0.44000821s) [upload=776889]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:16:00.857315054 (887h47m0.728398089s) [upload=776889]: Done uploading segments... Jan 11 15:16:00.857334662 (887h47m0.728417697s) [upload=776889]: Upload finished. err: <nil> Duration: 30.002649591s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXXXXXXXX.X.XX.XX...X] duration: 30.001324408s trace: https://jaeger.dev.storj.io/trace/29f06ff3cf95c6f5 FAIL (2026-01-11 15:16:19 UTC) orbiter warning: context deadline exceeded Duration: 30.001345579s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.XX] duration: 5.000951392s trace: https://jaeger.dev.storj.io/trace/70acd3f079b0c64e FAIL (2026-01-11 15:16:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001175499s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.494685288s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:16:02 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.494715705s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.X] duration: 5.001220848s trace: https://jaeger.dev.storj.io/trace/707197133fe26051 FAIL (2026-01-11 15:15:56 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001242369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.998091386s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:15:55 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.998119989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 5.75091125s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:15:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.750935486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX.] duration: 4.175653231s trace: https://jaeger.dev.storj.io/trace/6f517fa3ec09b4bc FAIL (2026-01-11 15:15:32 UTC) orbiter warning: context deadline exceeded Duration: 4.175704769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.573929515s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:15:23 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.573953099s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.96901539s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:15:22 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1.969056499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXXXXXXXX.X.XX.XX...] duration: 28.008400384s trace: https://jaeger.dev.storj.io/trace/1e54535df0d920d9 FAIL (2026-01-11 15:15:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.008416363s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..X.................................X...] duration: 5.833846802s trace: https://jaeger.dev.storj.io/trace/70d62a2a64e9256c FAIL (2026-01-11 15:15:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:13:07.350117387 (887h44m7.221200442s) [upload=776845]: Starting upload Jan 11 15:13:07.350210994 (887h44m7.221294039s) [upload=776845]: Done starting upload Jan 11 15:13:07.350333338 (887h44m7.221416383s) [upload=776845]: Uploading segments... Jan 11 15:13:07.350497631 (887h44m7.221580676s) [upload=776845]: Got next segment. Inline: false Jan 11 15:13:07.350555310 (887h44m7.221638355s) [upload=776845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:13:07.350561071 (887h44m7.221644106s) [upload=776845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:13:07.600923272 (887h44m7.472006307s) [seg_pos={0 0}, upload=776845]: Begin upload segment... Jan 11 15:13:07.600976232 (887h44m7.472059277s) [seg_pos={0 0}, upload=776845]: Done begin upload segment. Jan 11 15:13:07.601006239 (887h44m7.472089274s) [noise=true , address=145.79.227.40:4051 , piece_id=H5OCSK4FYZAG3WCNNRKE2FPMZZC2HSLJ7XSVRBAL67TMD2ORY6PQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601088416 (887h44m7.472171451s) [noise=true , address=145.79.227.16:4910 , piece_id=AMEREZ2G2ETQFX2ENJXFXDF7BUOMD3W7ADIU6TJ6Y6WT4VIQZ2GQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601609509 (887h44m7.472692554s) [noise=true , address=145.79.227.2:4114 , piece_id=TCYBZRSJ75FEB6Q7JBTXIR5JWAOATBFIM7MXY32BNKDSCMC2GKAA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601957682 (887h44m7.473040717s) [noise=true , address=145.79.227.29:4941 , piece_id=WM2Q7DUBRZ75YTF6VYUB4J6HUME4ZVB73P24LM7SHJJ4VFICOUPA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602267991 (887h44m7.473351027s) [noise=true , address=145.79.227.45:4230 , piece_id=CAQJBTOTGLHMVHDNMICJDRHI7AONRLAPTV4CX5Q6X5XCLE7VPQHA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602550911 (887h44m7.473633946s) [noise=true , address=145.79.227.27:4691 , piece_id=SHSFYUSX77MTIY3RFX5DWR437LJD27G4QSDCPP3RZDMI2B6RNAWA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602816296 (887h44m7.473899331s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.603170039 (887h44m7.474253074s) [noise=true , address=77.90.31.12:4589 , piece_id=CZ3WYVMEA5JUFCQO3N2VVP2UHWCETFGQU5SS6T5ZINY474MIOPUA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.603481162 (887h44m7.474564207s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.604325400 (887h44m7.475408435s) [noise=true , address=45.91.116.8:4045 , piece_id=22SXZ7TK6FODDEWSNZIWWPNAPY3ZLL2SMP7XALDDDCVAFNKJN6XQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.604988513 (887h44m7.476071548s) [noise=true , address=145.79.227.51:4411 , piece_id=I6JVQQDEIBBJFOMFXIQR3JA7VKKY2OXLDSC4TQMPLMBEQFZH7PRQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605302200 (887h44m7.476385235s) [noise=true , address=145.79.227.22:4511 , piece_id=DOGMFORDYGDQ7KOAC4XWI3EZFBGNGCTWRG6RJSD375W5344W67IQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605674869 (887h44m7.476757904s) [noise=true , address=145.79.227.43:4158 , piece_id=LQF3LKUPWVIT3NGIWCOAEOQBJCI25RFF52R5J4HCHNPDAPIS262A, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605988406 (887h44m7.477071441s) [noise=true , address=145.79.227.52:4445 , piece_id=RD4JUCKFLJGT5B2E7YDJEAWOFJSVI5ICYXYFC375VJADCGNDR6GA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.606276876 (887h44m7.477359911s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.606676657 (887h44m7.477759702s) [noise=true , address=145.79.227.49:4338 , piece_id=T75HFCURYDGB7ER744U7WEVKWHGO4SYQBZOOA3MQ2U2GWNH3M5WA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:10.397136698 (887h44m10.268219743s) [upload=776845]: Next returned nil segment Jan 11 15:13:10.397154301 (887h44m10.268237336s) [upload=776845]: Waiting for error group managing segments... Jan 11 15:13:10.861029296 (887h44m10.732112331s) [noise=true , address=145.79.227.29:4941 , piece_id=WM2Q7DUBRZ75YTF6VYUB4J6HUME4ZVB73P24LM7SHJJ4VFICOUPA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:10.980475396 (887h44m10.851558441s) [noise=true , address=45.91.116.8:4045 , piece_id=22SXZ7TK6FODDEWSNZIWWPNAPY3ZLL2SMP7XALDDDCVAFNKJN6XQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.490613767 (887h44m12.361696812s) [noise=true , address=77.90.31.12:4589 , piece_id=CZ3WYVMEA5JUFCQO3N2VVP2UHWCETFGQU5SS6T5ZINY474MIOPUA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.628139232 (887h44m12.499222267s) [noise=true , address=145.79.227.51:4411 , piece_id=I6JVQQDEIBBJFOMFXIQR3JA7VKKY2OXLDSC4TQMPLMBEQFZH7PRQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.790419032 (887h44m12.661502067s) [noise=true , address=145.79.227.49:4338 , piece_id=T75HFCURYDGB7ER744U7WEVKWHGO4SYQBZOOA3MQ2U2GWNH3M5WA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:13.589901957 (887h44m13.460985002s) [noise=true , address=145.79.227.16:4910 , piece_id=AMEREZ2G2ETQFX2ENJXFXDF7BUOMD3W7ADIU6TJ6Y6WT4VIQZ2GQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:14.019031442 (887h44m13.890114487s) [noise=true , address=145.79.227.52:4445 , piece_id=RD4JUCKFLJGT5B2E7YDJEAWOFJSVI5ICYXYFC375VJADCGNDR6GA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:14.715997822 (887h44m14.587080857s) [noise=true , address=145.79.227.27:4691 , piece_id=SHSFYUSX77MTIY3RFX5DWR437LJD27G4QSDCPP3RZDMI2B6RNAWA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.058750581 (887h44m15.929833626s) [noise=true , address=145.79.227.43:4158 , piece_id=LQF3LKUPWVIT3NGIWCOAEOQBJCI25RFF52R5J4HCHNPDAPIS262A, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.165005102 (887h44m16.036088137s) [noise=true , address=145.79.227.2:4114 , piece_id=TCYBZRSJ75FEB6Q7JBTXIR5JWAOATBFIM7MXY32BNKDSCMC2GKAA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.805830834 (887h44m16.676913879s) [noise=true , address=145.79.227.40:4051 , piece_id=H5OCSK4FYZAG3WCNNRKE2FPMZZC2HSLJ7XSVRBAL67TMD2ORY6PQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:19.502038932 (887h44m19.373121967s) [noise=true , address=145.79.227.45:4230 , piece_id=CAQJBTOTGLHMVHDNMICJDRHI7AONRLAPTV4CX5Q6X5XCLE7VPQHA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:22.404086100 (887h44m22.275169135s) [noise=true , address=145.79.227.22:4511 , piece_id=DOGMFORDYGDQ7KOAC4XWI3EZFBGNGCTWRG6RJSD375W5344W67IQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:22.404105858 (887h44m22.275188894s) [seg_pos={0 0}, upload=776845]: Segment reached optimal threshold of 13 pieces. Jan 11 15:13:22.404377606 (887h44m22.275460641s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 11 15:13:22.404386954 (887h44m22.275469989s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404426479 (887h44m22.275509514s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Jan 11 15:13:22.404442600 (887h44m22.275525635s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404530707 (887h44m22.275613742s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: read tcp 188.166.225.170:34532->145.79.227.44:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:13:22.404536678 (887h44m22.275619714s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404596312 (887h44m22.275679347s) [seg_pos={0 0}, upload=776845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:13:22.404711551 (887h44m22.275794586s) [upload=776845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:13:22.404716933 (887h44m22.275799978s) [upload=776845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:13:22.700402434 (887h44m22.571485479s) [upload=776845]: Done uploading segments... Jan 11 15:13:22.700421460 (887h44m22.571504495s) [upload=776845]: Upload finished. err: <nil> Duration: 5.833873444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X..X.XX..X.........XX...X....XX.X..X......X.X...............] duration: 19.235831356s trace: https://jaeger.dev.storj.io/trace/3beb10e6790340c6 FAIL (2026-01-11 15:15:11 UTC) orbiter warning: context deadline exceeded Duration: 19.235926327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 5.973121057s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:15:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 5.973178316s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXXX] duration: 5.0014829s trace: https://jaeger.dev.storj.io/trace/6f517fa3ec09b4bc FAIL (2026-01-11 15:14:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001522085s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 43 out of 60 is failed (threshold=3) [.X.XXXXX..XXX.X.XX....XXX.XXXXXXXXXXXXXX.XXXXXXXXX.X.XX.XX..] duration: 28.359472922s trace: https://jaeger.dev.storj.io/trace/1e54535df0d920d9 FAIL (2026-01-11 15:14:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.359496218s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XX..X.XX..X.........XX...X....XX.X..X......X.X..............] duration: 35.08877835s trace: https://jaeger.dev.storj.io/trace/3beb10e6790340c6 FAIL (2026-01-11 15:14:43 UTC) orbiter warning: context deadline exceeded Duration: 35.088803449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..X.................................X..] duration: 5.770650958s trace: https://jaeger.dev.storj.io/trace/70d62a2a64e9256c FAIL (2026-01-11 15:14:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:13:07.350117387 (887h44m7.221200442s) [upload=776845]: Starting upload Jan 11 15:13:07.350210994 (887h44m7.221294039s) [upload=776845]: Done starting upload Jan 11 15:13:07.350333338 (887h44m7.221416383s) [upload=776845]: Uploading segments... Jan 11 15:13:07.350497631 (887h44m7.221580676s) [upload=776845]: Got next segment. Inline: false Jan 11 15:13:07.350555310 (887h44m7.221638355s) [upload=776845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:13:07.350561071 (887h44m7.221644106s) [upload=776845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:13:07.600923272 (887h44m7.472006307s) [seg_pos={0 0}, upload=776845]: Begin upload segment... Jan 11 15:13:07.600976232 (887h44m7.472059277s) [seg_pos={0 0}, upload=776845]: Done begin upload segment. Jan 11 15:13:07.601006239 (887h44m7.472089274s) [noise=true , address=145.79.227.40:4051 , piece_id=H5OCSK4FYZAG3WCNNRKE2FPMZZC2HSLJ7XSVRBAL67TMD2ORY6PQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601088416 (887h44m7.472171451s) [noise=true , address=145.79.227.16:4910 , piece_id=AMEREZ2G2ETQFX2ENJXFXDF7BUOMD3W7ADIU6TJ6Y6WT4VIQZ2GQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601609509 (887h44m7.472692554s) [noise=true , address=145.79.227.2:4114 , piece_id=TCYBZRSJ75FEB6Q7JBTXIR5JWAOATBFIM7MXY32BNKDSCMC2GKAA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601957682 (887h44m7.473040717s) [noise=true , address=145.79.227.29:4941 , piece_id=WM2Q7DUBRZ75YTF6VYUB4J6HUME4ZVB73P24LM7SHJJ4VFICOUPA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602267991 (887h44m7.473351027s) [noise=true , address=145.79.227.45:4230 , piece_id=CAQJBTOTGLHMVHDNMICJDRHI7AONRLAPTV4CX5Q6X5XCLE7VPQHA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602550911 (887h44m7.473633946s) [noise=true , address=145.79.227.27:4691 , piece_id=SHSFYUSX77MTIY3RFX5DWR437LJD27G4QSDCPP3RZDMI2B6RNAWA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602816296 (887h44m7.473899331s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.603170039 (887h44m7.474253074s) [noise=true , address=77.90.31.12:4589 , piece_id=CZ3WYVMEA5JUFCQO3N2VVP2UHWCETFGQU5SS6T5ZINY474MIOPUA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.603481162 (887h44m7.474564207s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.604325400 (887h44m7.475408435s) [noise=true , address=45.91.116.8:4045 , piece_id=22SXZ7TK6FODDEWSNZIWWPNAPY3ZLL2SMP7XALDDDCVAFNKJN6XQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.604988513 (887h44m7.476071548s) [noise=true , address=145.79.227.51:4411 , piece_id=I6JVQQDEIBBJFOMFXIQR3JA7VKKY2OXLDSC4TQMPLMBEQFZH7PRQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605302200 (887h44m7.476385235s) [noise=true , address=145.79.227.22:4511 , piece_id=DOGMFORDYGDQ7KOAC4XWI3EZFBGNGCTWRG6RJSD375W5344W67IQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605674869 (887h44m7.476757904s) [noise=true , address=145.79.227.43:4158 , piece_id=LQF3LKUPWVIT3NGIWCOAEOQBJCI25RFF52R5J4HCHNPDAPIS262A, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605988406 (887h44m7.477071441s) [noise=true , address=145.79.227.52:4445 , piece_id=RD4JUCKFLJGT5B2E7YDJEAWOFJSVI5ICYXYFC375VJADCGNDR6GA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.606276876 (887h44m7.477359911s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.606676657 (887h44m7.477759702s) [noise=true , address=145.79.227.49:4338 , piece_id=T75HFCURYDGB7ER744U7WEVKWHGO4SYQBZOOA3MQ2U2GWNH3M5WA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:10.397136698 (887h44m10.268219743s) [upload=776845]: Next returned nil segment Jan 11 15:13:10.397154301 (887h44m10.268237336s) [upload=776845]: Waiting for error group managing segments... Jan 11 15:13:10.861029296 (887h44m10.732112331s) [noise=true , address=145.79.227.29:4941 , piece_id=WM2Q7DUBRZ75YTF6VYUB4J6HUME4ZVB73P24LM7SHJJ4VFICOUPA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:10.980475396 (887h44m10.851558441s) [noise=true , address=45.91.116.8:4045 , piece_id=22SXZ7TK6FODDEWSNZIWWPNAPY3ZLL2SMP7XALDDDCVAFNKJN6XQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.490613767 (887h44m12.361696812s) [noise=true , address=77.90.31.12:4589 , piece_id=CZ3WYVMEA5JUFCQO3N2VVP2UHWCETFGQU5SS6T5ZINY474MIOPUA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.628139232 (887h44m12.499222267s) [noise=true , address=145.79.227.51:4411 , piece_id=I6JVQQDEIBBJFOMFXIQR3JA7VKKY2OXLDSC4TQMPLMBEQFZH7PRQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.790419032 (887h44m12.661502067s) [noise=true , address=145.79.227.49:4338 , piece_id=T75HFCURYDGB7ER744U7WEVKWHGO4SYQBZOOA3MQ2U2GWNH3M5WA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:13.589901957 (887h44m13.460985002s) [noise=true , address=145.79.227.16:4910 , piece_id=AMEREZ2G2ETQFX2ENJXFXDF7BUOMD3W7ADIU6TJ6Y6WT4VIQZ2GQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:14.019031442 (887h44m13.890114487s) [noise=true , address=145.79.227.52:4445 , piece_id=RD4JUCKFLJGT5B2E7YDJEAWOFJSVI5ICYXYFC375VJADCGNDR6GA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:14.715997822 (887h44m14.587080857s) [noise=true , address=145.79.227.27:4691 , piece_id=SHSFYUSX77MTIY3RFX5DWR437LJD27G4QSDCPP3RZDMI2B6RNAWA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.058750581 (887h44m15.929833626s) [noise=true , address=145.79.227.43:4158 , piece_id=LQF3LKUPWVIT3NGIWCOAEOQBJCI25RFF52R5J4HCHNPDAPIS262A, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.165005102 (887h44m16.036088137s) [noise=true , address=145.79.227.2:4114 , piece_id=TCYBZRSJ75FEB6Q7JBTXIR5JWAOATBFIM7MXY32BNKDSCMC2GKAA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.805830834 (887h44m16.676913879s) [noise=true , address=145.79.227.40:4051 , piece_id=H5OCSK4FYZAG3WCNNRKE2FPMZZC2HSLJ7XSVRBAL67TMD2ORY6PQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:19.502038932 (887h44m19.373121967s) [noise=true , address=145.79.227.45:4230 , piece_id=CAQJBTOTGLHMVHDNMICJDRHI7AONRLAPTV4CX5Q6X5XCLE7VPQHA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:22.404086100 (887h44m22.275169135s) [noise=true , address=145.79.227.22:4511 , piece_id=DOGMFORDYGDQ7KOAC4XWI3EZFBGNGCTWRG6RJSD375W5344W67IQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:22.404105858 (887h44m22.275188894s) [seg_pos={0 0}, upload=776845]: Segment reached optimal threshold of 13 pieces. Jan 11 15:13:22.404377606 (887h44m22.275460641s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 11 15:13:22.404386954 (887h44m22.275469989s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404426479 (887h44m22.275509514s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Jan 11 15:13:22.404442600 (887h44m22.275525635s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404530707 (887h44m22.275613742s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: read tcp 188.166.225.170:34532->145.79.227.44:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:13:22.404536678 (887h44m22.275619714s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404596312 (887h44m22.275679347s) [seg_pos={0 0}, upload=776845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:13:22.404711551 (887h44m22.275794586s) [upload=776845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:13:22.404716933 (887h44m22.275799978s) [upload=776845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:13:22.700402434 (887h44m22.571485479s) [upload=776845]: Done uploading segments... Jan 11 15:13:22.700421460 (887h44m22.571504495s) [upload=776845]: Upload finished. err: <nil> Duration: 5.770681607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.635034388s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:14:27 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 5.635070758s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.072225616s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:14:12 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 2.07229601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 22.826955314s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:14:10 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 22.82697393s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001346308s trace: https://jaeger.dev.storj.io/trace/c31ec8b6bb524c FAIL (2026-01-11 15:14:10 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1m0.001368431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 12.408994342s trace: https://jaeger.dev.storj.io/trace/122156f6a682ae8a FAIL (2026-01-11 15:14:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ce_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.300519664 (887h19m52.171602699s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301064591 (887h19m52.172147626s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301431329 (887h19m52.172514364s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.301886867 (887h19m52.172969902s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302170777 (887h19m52.173253812s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302221844 (887h19m52.173304879s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302628739 (887h19m52.173711774s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.302736915 (887h19m52.17381996s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303009564 (887h19m52.174092609s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303457567 (887h19m52.174540613s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303477966 (887h19m52.174561001s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303539132 (887h19m52.174622167s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.303957589 (887h19m52.175040624s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304147832 (887h19m52.175230867s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.304943848 (887h19m52.176026893s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:52.305472424 (887h19m52.176555459s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Uploading piece... Jan 11 14:48:53.758274050 (887h19m53.629357085s) [upload=776497]: Next returned nil segment Jan 11 14:48:53.758286724 (887h19m53.629369759s) [upload=776497]: Waiting for error group managing segments... Jan 11 14:48:54.165049915 (887h19m54.03613298s) [noise=true , address=64.49.13.169:4714 , piece_id=24OFMWJXEQEOIWEDVM56ZMNCXLS5MQDEKSVCYH2J2D6GHEXTFPPA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.012423457 (887h19m54.883506492s) [noise=true , address=193.149.164.143:4701 , piece_id=XFGIKVHPAMWCCXSZ2IBGRKSCD6ANE4MCXT5WQ5X4Q32RPDENXPUQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:55.765555640 (887h19m55.636638685s) [noise=true , address=77.90.31.10:4528 , piece_id=7I2O62ME42ANAPUOJ55A3VSWCWPCXUC52QY73JZYVYPYZPDVRQGQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.384302106 (887h19m56.255385141s) [noise=true , address=145.79.227.40:4069 , piece_id=XDVP7RLX7YV4OADUO53KV46WAG4FYPWD3MBSU3MJCZFIX55VXDLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.591221818 (887h19m56.462304853s) [noise=true , address=45.91.116.5:4322 , piece_id=7D7ZUGF5DGERAIKRKJNTX7VFRXZECQBNY3OKS5DIVK5RINZDSCZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:56.855036076 (887h19m56.726119111s) [noise=true , address=145.79.227.30:4787 , piece_id=G2A6ICN4VWPLI5LMI6TTL56OHBEADHGF6D7E2Z5JD776ELOX4SKQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.447733941 (887h19m57.318816976s) [noise=true , address=45.91.116.10:4111 , piece_id=T3WD6WGGS2A6WMKOVAUFELN23MD7I76PZSPCX347V3HOTZ6L4XFA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:57.841848387 (887h19m57.712931422s) [noise=true , address=45.91.116.9:4293 , piece_id=64G5GA4RCNFIDVICICK36OA4DNE4APKBQPITDTWQZADJ5RCWTQYA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.202209371 (887h19m58.073292406s) [noise=true , address=77.90.31.18:4828 , piece_id=RUM7VMBRHJQAN2WQKWPKFJDUNOCWAKMD4IH6ISQAT4APZO6SPDIQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:58.987951347 (887h19m58.859034382s) [noise=true , address=77.90.31.14:4687 , piece_id=4FM2JITE3KA2VMV5B2NG4M5S7QFF6ARTM4VFJWKLCGLUMHAYYW7A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:48:59.587019872 (887h19m59.458102917s) [noise=true , address=145.79.227.49:4366 , piece_id=T5GC443V5AT23BWSQR5SROWWM2SQHF6VGABOED2IG3SPW3JYSPJQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:00.486336099 (887h20m0.357419144s) [noise=true , address=145.79.227.34:4939 , piece_id=5FQSIGVGWFTTMLQX67BXHGEASYZOQCG6UNOWYEIIUTBY7LZ24G2A, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:01.681374018 (887h20m1.552457073s) [noise=true , address=77.90.31.15:4709 , piece_id=U5TBMLYNAJBNJGHRD4QNKNGEZYX53DRINLMEALOMQVYF4C3AF6PA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662257911 (887h20m2.533340956s) [noise=true , address=45.91.116.11:4127 , piece_id=4DGVPZBMONTZKZ6MZUAH2YYTZO6CQ5WDQVCI4D7B6WC5PX3AH4WQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: <nil> Jan 11 14:49:02.662281735 (887h20m2.53336478s) [seg_pos={0 0}, upload=776497]: Segment reached optimal threshold of 14 pieces. Jan 11 14:49:02.662643675 (887h20m2.53372672s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.662654024 (887h20m2.53373707s) [noise=true , address=145.79.227.33:4891 , piece_id=QWYGFCU6NIZZAWKAQ2CJ6CTNNDZQUY4DPS4GKQQJVTOMYXM624ZQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.662986458 (887h20m2.534069503s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JfhuZrWmNr3sBwxV9DKCs3CNQmeGEYnNN1PLwNtAE1dN4J77i): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663000885 (887h20m2.53408393s) [noise=true , address=145.79.227.52:4448 , piece_id=C4FZTOCO77R76LSHKBH5EZFNWAYEIZGFBS7EU66WZRM65RL6DXLA, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663013579 (887h20m2.534096624s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126J5HU5z7HfKE1XJfqxcz3qUsPJod2M8z3Ckg3MGYnAPD4fRHK): protocol: expected piece hash; context canceled; EOF Jan 11 14:49:02.663022006 (887h20m2.534105051s) [noise=true , address=45.91.116.7:4042 , piece_id=DJUHAPUH555QN4EN5N3Z5PJLRKLEWS6MWBS7RC4BHGI5KC75VAHQ, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663366712 (887h20m2.534449767s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1veqoBGw6a8AGMoigyuNJMjcEvCT93xRq2B1d2KNhs6aEmSqVF): protocol: read tcp 188.166.225.170:57032->45.91.116.6:4003: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 14:49:02.663376571 (887h20m2.534459616s) [noise=true , address=45.91.116.6:4003 , piece_id=KHRBDGBVEGFDV3B3BDPMN32T3LGWH2UJTO6O6HE64MKZWBCLXB3Q, seg_pos={0 0}, upload=776497]: Long tail cancellation due to optimal threshold Jan 11 14:49:02.663395456 (887h20m2.534478491s) [seg_pos={0 0}, upload=776497]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 11 14:49:02.663519342 (887h20m2.534602377s) [upload=776497]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 14:49:02.663525824 (887h20m2.53460887s) [upload=776497]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 14:49:02.957663782 (887h20m2.828746827s) [upload=776497]: Done uploading segments... Jan 11 14:49:02.957689441 (887h20m2.828772497s) [upload=776497]: Upload finished. err: <nil> Duration: 12.409019368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..X.................................X.] duration: 8.433105513s trace: https://jaeger.dev.storj.io/trace/70d62a2a64e9256c FAIL (2026-01-11 15:14:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 11 15:13:07.350117387 (887h44m7.221200442s) [upload=776845]: Starting upload Jan 11 15:13:07.350210994 (887h44m7.221294039s) [upload=776845]: Done starting upload Jan 11 15:13:07.350333338 (887h44m7.221416383s) [upload=776845]: Uploading segments... Jan 11 15:13:07.350497631 (887h44m7.221580676s) [upload=776845]: Got next segment. Inline: false Jan 11 15:13:07.350555310 (887h44m7.221638355s) [upload=776845]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 11 15:13:07.350561071 (887h44m7.221644106s) [upload=776845]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 11 15:13:07.600923272 (887h44m7.472006307s) [seg_pos={0 0}, upload=776845]: Begin upload segment... Jan 11 15:13:07.600976232 (887h44m7.472059277s) [seg_pos={0 0}, upload=776845]: Done begin upload segment. Jan 11 15:13:07.601006239 (887h44m7.472089274s) [noise=true , address=145.79.227.40:4051 , piece_id=H5OCSK4FYZAG3WCNNRKE2FPMZZC2HSLJ7XSVRBAL67TMD2ORY6PQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601088416 (887h44m7.472171451s) [noise=true , address=145.79.227.16:4910 , piece_id=AMEREZ2G2ETQFX2ENJXFXDF7BUOMD3W7ADIU6TJ6Y6WT4VIQZ2GQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601609509 (887h44m7.472692554s) [noise=true , address=145.79.227.2:4114 , piece_id=TCYBZRSJ75FEB6Q7JBTXIR5JWAOATBFIM7MXY32BNKDSCMC2GKAA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.601957682 (887h44m7.473040717s) [noise=true , address=145.79.227.29:4941 , piece_id=WM2Q7DUBRZ75YTF6VYUB4J6HUME4ZVB73P24LM7SHJJ4VFICOUPA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602267991 (887h44m7.473351027s) [noise=true , address=145.79.227.45:4230 , piece_id=CAQJBTOTGLHMVHDNMICJDRHI7AONRLAPTV4CX5Q6X5XCLE7VPQHA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602550911 (887h44m7.473633946s) [noise=true , address=145.79.227.27:4691 , piece_id=SHSFYUSX77MTIY3RFX5DWR437LJD27G4QSDCPP3RZDMI2B6RNAWA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.602816296 (887h44m7.473899331s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.603170039 (887h44m7.474253074s) [noise=true , address=77.90.31.12:4589 , piece_id=CZ3WYVMEA5JUFCQO3N2VVP2UHWCETFGQU5SS6T5ZINY474MIOPUA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.603481162 (887h44m7.474564207s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.604325400 (887h44m7.475408435s) [noise=true , address=45.91.116.8:4045 , piece_id=22SXZ7TK6FODDEWSNZIWWPNAPY3ZLL2SMP7XALDDDCVAFNKJN6XQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.604988513 (887h44m7.476071548s) [noise=true , address=145.79.227.51:4411 , piece_id=I6JVQQDEIBBJFOMFXIQR3JA7VKKY2OXLDSC4TQMPLMBEQFZH7PRQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605302200 (887h44m7.476385235s) [noise=true , address=145.79.227.22:4511 , piece_id=DOGMFORDYGDQ7KOAC4XWI3EZFBGNGCTWRG6RJSD375W5344W67IQ, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605674869 (887h44m7.476757904s) [noise=true , address=145.79.227.43:4158 , piece_id=LQF3LKUPWVIT3NGIWCOAEOQBJCI25RFF52R5J4HCHNPDAPIS262A, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.605988406 (887h44m7.477071441s) [noise=true , address=145.79.227.52:4445 , piece_id=RD4JUCKFLJGT5B2E7YDJEAWOFJSVI5ICYXYFC375VJADCGNDR6GA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.606276876 (887h44m7.477359911s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:07.606676657 (887h44m7.477759702s) [noise=true , address=145.79.227.49:4338 , piece_id=T75HFCURYDGB7ER744U7WEVKWHGO4SYQBZOOA3MQ2U2GWNH3M5WA, seg_pos={0 0}, upload=776845]: Uploading piece... Jan 11 15:13:10.397136698 (887h44m10.268219743s) [upload=776845]: Next returned nil segment Jan 11 15:13:10.397154301 (887h44m10.268237336s) [upload=776845]: Waiting for error group managing segments... Jan 11 15:13:10.861029296 (887h44m10.732112331s) [noise=true , address=145.79.227.29:4941 , piece_id=WM2Q7DUBRZ75YTF6VYUB4J6HUME4ZVB73P24LM7SHJJ4VFICOUPA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:10.980475396 (887h44m10.851558441s) [noise=true , address=45.91.116.8:4045 , piece_id=22SXZ7TK6FODDEWSNZIWWPNAPY3ZLL2SMP7XALDDDCVAFNKJN6XQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.490613767 (887h44m12.361696812s) [noise=true , address=77.90.31.12:4589 , piece_id=CZ3WYVMEA5JUFCQO3N2VVP2UHWCETFGQU5SS6T5ZINY474MIOPUA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.628139232 (887h44m12.499222267s) [noise=true , address=145.79.227.51:4411 , piece_id=I6JVQQDEIBBJFOMFXIQR3JA7VKKY2OXLDSC4TQMPLMBEQFZH7PRQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:12.790419032 (887h44m12.661502067s) [noise=true , address=145.79.227.49:4338 , piece_id=T75HFCURYDGB7ER744U7WEVKWHGO4SYQBZOOA3MQ2U2GWNH3M5WA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:13.589901957 (887h44m13.460985002s) [noise=true , address=145.79.227.16:4910 , piece_id=AMEREZ2G2ETQFX2ENJXFXDF7BUOMD3W7ADIU6TJ6Y6WT4VIQZ2GQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:14.019031442 (887h44m13.890114487s) [noise=true , address=145.79.227.52:4445 , piece_id=RD4JUCKFLJGT5B2E7YDJEAWOFJSVI5ICYXYFC375VJADCGNDR6GA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:14.715997822 (887h44m14.587080857s) [noise=true , address=145.79.227.27:4691 , piece_id=SHSFYUSX77MTIY3RFX5DWR437LJD27G4QSDCPP3RZDMI2B6RNAWA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.058750581 (887h44m15.929833626s) [noise=true , address=145.79.227.43:4158 , piece_id=LQF3LKUPWVIT3NGIWCOAEOQBJCI25RFF52R5J4HCHNPDAPIS262A, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.165005102 (887h44m16.036088137s) [noise=true , address=145.79.227.2:4114 , piece_id=TCYBZRSJ75FEB6Q7JBTXIR5JWAOATBFIM7MXY32BNKDSCMC2GKAA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:16.805830834 (887h44m16.676913879s) [noise=true , address=145.79.227.40:4051 , piece_id=H5OCSK4FYZAG3WCNNRKE2FPMZZC2HSLJ7XSVRBAL67TMD2ORY6PQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:19.502038932 (887h44m19.373121967s) [noise=true , address=145.79.227.45:4230 , piece_id=CAQJBTOTGLHMVHDNMICJDRHI7AONRLAPTV4CX5Q6X5XCLE7VPQHA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:22.404086100 (887h44m22.275169135s) [noise=true , address=145.79.227.22:4511 , piece_id=DOGMFORDYGDQ7KOAC4XWI3EZFBGNGCTWRG6RJSD375W5344W67IQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: <nil> Jan 11 15:13:22.404105858 (887h44m22.275188894s) [seg_pos={0 0}, upload=776845]: Segment reached optimal threshold of 13 pieces. Jan 11 15:13:22.404377606 (887h44m22.275460641s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Jan 11 15:13:22.404386954 (887h44m22.275469989s) [noise=true , address=145.79.227.50:4373 , piece_id=QCTVQEY7EABWNJZMZWFJ5ROSFCT64UMX5V5M6RKLGEBY7XCNMWSQ, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404426479 (887h44m22.275509514s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8): protocol: expected piece hash; context canceled; EOF Jan 11 15:13:22.404442600 (887h44m22.275525635s) [noise=true , address=145.79.227.38:4977 , piece_id=Y4LWP7ZNLJOFYICB5KZ65K2PBU47YMYC6YFTCQNGFUFFCD6B3ZAA, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404530707 (887h44m22.275613742s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: read tcp 188.166.225.170:34532->145.79.227.44:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 11 15:13:22.404536678 (887h44m22.275619714s) [noise=true , address=145.79.227.44:4195 , piece_id=JBXTHNP6G3LF4U6UFYLBGZYGBUBCCRRY2QM5VWOU6JPXOFJXBPNA, seg_pos={0 0}, upload=776845]: Long tail cancellation due to optimal threshold Jan 11 15:13:22.404596312 (887h44m22.275679347s) [seg_pos={0 0}, upload=776845]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 11 15:13:22.404711551 (887h44m22.275794586s) [upload=776845]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 11 15:13:22.404716933 (887h44m22.275799978s) [upload=776845]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 11 15:13:22.700402434 (887h44m22.571485479s) [upload=776845]: Done uploading segments... Jan 11 15:13:22.700421460 (887h44m22.571504495s) [upload=776845]: Upload finished. err: <nil> Duration: 8.433131713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X.XXXXXXXXXXXX.XXXX] duration: 5.000642396s trace: https://jaeger.dev.storj.io/trace/6f05001f436e081f FAIL (2026-01-11 15:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000670009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.XX..X.XX..X.........XX...X....XX.X..X......X.X.............] duration: 23.639441506s trace: https://jaeger.dev.storj.io/trace/3beb10e6790340c6 FAIL (2026-01-11 15:14:01 UTC) orbiter warning: context deadline exceeded Duration: 23.63947517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005679901s trace: https://jaeger.dev.storj.io/trace/7fd32fc43d58b31c FAIL (2026-01-11 15:13:47 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 30.005704297s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)