Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 44.816863455s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:11:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 44.816893763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.24822115s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:11:43 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.248251038s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX..] duration: 4.815534094s trace: https://jaeger.dev.storj.io/trace/52f4e8411dc99da4 FAIL (2025-12-28 05:11:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.815558132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.462796096s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:11:20 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.462831154s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X.] duration: 6.00170694s trace: https://jaeger.dev.storj.io/trace/85753c584f83639 FAIL (2025-12-28 05:11:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: FOG6ONI3BAUQVLOQ6SPIM5JO2VHV7UNZBS3X6DEJ35MJDN73ED2Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SA55AB5C3PCLPMXQXY3X5IWZMYUKO22O2ETASQKSHJ54I6NXE5XA) context deadline exceeded ======LOGS====== Dec 28 05:09:57.879931513 (541h40m57.751014559s) [upload=475023]: Starting upload Dec 28 05:09:57.880068214 (541h40m57.751151249s) [upload=475023]: Done starting upload Dec 28 05:09:57.880161321 (541h40m57.751244366s) [upload=475023]: Uploading segments... Dec 28 05:09:57.880356253 (541h40m57.751439308s) [upload=475023]: Got next segment. Inline: false Dec 28 05:09:57.880420685 (541h40m57.75150373s) [upload=475023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:09:57.880427819 (541h40m57.751510854s) [upload=475023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:09:58.125470970 (541h40m57.996554005s) [seg_pos={0 0}, upload=475023]: Begin upload segment... Dec 28 05:09:58.125530965 (541h40m57.996614s) [seg_pos={0 0}, upload=475023]: Done begin upload segment. Dec 28 05:09:58.125641605 (541h40m57.99672465s) [noise=true , address=145.79.227.18:4006 , piece_id=DNTDZAVTGE7PV5NK26RNVWA7HZB3QBKHD5YCRAETKT7IC7HUSQRA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.126759634 (541h40m57.997842679s) [noise=true , address=145.79.227.31:4798 , piece_id=3CE5D7JHSNWABTEM67E2YNFVHIAR67DRK5YCVKAN7WC6RUTVTYSQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.127371480 (541h40m57.998454515s) [noise=true , address=145.79.227.30:4761 , piece_id=OQ5P6IQ7GXIT7RJEHCX4IYD4F2AHGRUQ5OW4RMLNAAW76TMFFCFQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.127795106 (541h40m57.998878151s) [noise=true , address=145.79.227.41:4085 , piece_id=QCK2FSJR5DPSJMMAQ5IY3JXQHEQMU44ZJ7OWMK4G7V2YTJ3G2YWQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.128191491 (541h40m57.999274536s) [noise=true , address=77.90.31.11:4555 , piece_id=62XIEVBRPTY2N2QIEL5G7P4FK7KMDZFZUF3OZFX6JEU7QVUOHP4A, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.128603285 (541h40m57.99968632s) [noise=true , address=45.91.116.9:4261 , piece_id=JS6QODZKG56DD3QJHLUFUMJY4VJ3GMNZTSLETRE6C5BEBI3ZPCOA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.129014939 (541h40m58.000097974s) [noise=true , address=45.91.116.5:4300 , piece_id=HNG6PXM2AHZNBZALM7DXFVRQQHWNFNWVEIS7HTQO76FSZTTSDM3A, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.129415281 (541h40m58.000498326s) [noise=true , address=145.79.227.10:4403 , piece_id=EZHCRWCAJXOGCXPQIMS2X5QAJFBOA5CQRESBV2OQV6G4I7NIMRZA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.129870668 (541h40m58.000953703s) [noise=true , address=145.79.227.5:4221 , piece_id=XDT6EWP3MY2BCUMIX4XHDS6Z7Q5XC536WRCJPXPIQTBKHE4W3VPQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.130269628 (541h40m58.001352663s) [noise=true , address=45.91.116.6:4974 , piece_id=FOG6ONI3BAUQVLOQ6SPIM5JO2VHV7UNZBS3X6DEJ35MJDN73ED2Q, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.130710467 (541h40m58.001793513s) [noise=true , address=145.79.227.13:4802 , piece_id=YUJFNRRF2U6DACZJD5PD5LSPE4NRVRHRN4OETMOS3BKZ3B62HORQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.131166777 (541h40m58.002249822s) [noise=true , address=145.79.227.16:4909 , piece_id=P4OGPUOTFN25C5MXKJNDUAXJOKUVN3UCHI6B5WYXCNRTU5OCBDRA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.132332706 (541h40m58.003415741s) [noise=true , address=145.79.227.2:4114 , piece_id=4ELMOELTYFMIUKVFTMNCLB2U5AAGESZJOQ4M54O7KDIWQZWZYVGQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.132924353 (541h40m58.004007398s) [noise=true , address=145.79.227.11:4731 , piece_id=L4QMR4UVQGJ3ME5A3UU3VHDWXGBS5CY6KRG36NRPDZZUUYXXRWAA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.133431489 (541h40m58.004514534s) [noise=true , address=77.90.31.10:4517 , piece_id=SA55AB5C3PCLPMXQXY3X5IWZMYUKO22O2ETASQKSHJ54I6NXE5XA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.134133205 (541h40m58.00521625s) [noise=true , address=145.79.227.15:4873 , piece_id=53G3AIBGMQOWFMXOFIGEH6MWTCSZDNOAYEJWBCQQJLCAEUPRKYEA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:59.864632552 (541h40m59.735715587s) [upload=475023]: Next returned nil segment Dec 28 05:09:59.864653692 (541h40m59.735736728s) [upload=475023]: Waiting for error group managing segments... Dec 28 05:10:00.154658578 (541h41m0.025741633s) [noise=true , address=145.79.227.5:4221 , piece_id=XDT6EWP3MY2BCUMIX4XHDS6Z7Q5XC536WRCJPXPIQTBKHE4W3VPQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.155028513 (541h41m0.026111548s) [noise=true , address=145.79.227.10:4403 , piece_id=EZHCRWCAJXOGCXPQIMS2X5QAJFBOA5CQRESBV2OQV6G4I7NIMRZA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.160894482 (541h41m0.031977527s) [noise=true , address=145.79.227.15:4873 , piece_id=53G3AIBGMQOWFMXOFIGEH6MWTCSZDNOAYEJWBCQQJLCAEUPRKYEA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.165427544 (541h41m0.036510589s) [noise=true , address=145.79.227.18:4006 , piece_id=DNTDZAVTGE7PV5NK26RNVWA7HZB3QBKHD5YCRAETKT7IC7HUSQRA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.173145761 (541h41m0.044228806s) [noise=true , address=145.79.227.31:4798 , piece_id=3CE5D7JHSNWABTEM67E2YNFVHIAR67DRK5YCVKAN7WC6RUTVTYSQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.185687532 (541h41m0.056770567s) [noise=true , address=145.79.227.13:4802 , piece_id=YUJFNRRF2U6DACZJD5PD5LSPE4NRVRHRN4OETMOS3BKZ3B62HORQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.191748122 (541h41m0.062831167s) [noise=true , address=145.79.227.16:4909 , piece_id=P4OGPUOTFN25C5MXKJNDUAXJOKUVN3UCHI6B5WYXCNRTU5OCBDRA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.322942437 (541h41m0.194025472s) [noise=true , address=77.90.31.11:4555 , piece_id=62XIEVBRPTY2N2QIEL5G7P4FK7KMDZFZUF3OZFX6JEU7QVUOHP4A, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.466260648 (541h41m0.337343693s) [noise=true , address=45.91.116.6:4974 , piece_id=FOG6ONI3BAUQVLOQ6SPIM5JO2VHV7UNZBS3X6DEJ35MJDN73ED2Q, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.496997125 (541h41m0.36808016s) [noise=true , address=45.91.116.5:4300 , piece_id=HNG6PXM2AHZNBZALM7DXFVRQQHWNFNWVEIS7HTQO76FSZTTSDM3A, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.700241796 (541h41m0.571324831s) [noise=true , address=77.90.31.10:4517 , piece_id=SA55AB5C3PCLPMXQXY3X5IWZMYUKO22O2ETASQKSHJ54I6NXE5XA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.763585372 (541h41m0.634668407s) [noise=true , address=45.91.116.9:4261 , piece_id=JS6QODZKG56DD3QJHLUFUMJY4VJ3GMNZTSLETRE6C5BEBI3ZPCOA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:01.098168231 (541h41m0.969251276s) [noise=true , address=145.79.227.41:4085 , piece_id=QCK2FSJR5DPSJMMAQ5IY3JXQHEQMU44ZJ7OWMK4G7V2YTJ3G2YWQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:01.098193119 (541h41m0.969276164s) [seg_pos={0 0}, upload=475023]: Segment reached optimal threshold of 13 pieces. Dec 28 05:10:01.098437224 (541h41m0.969520279s) [noise=true , address=145.79.227.30:4761 , piece_id=OQ5P6IQ7GXIT7RJEHCX4IYD4F2AHGRUQ5OW4RMLNAAW76TMFFCFQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:10:01.098448094 (541h41m0.969531139s) [noise=true , address=145.79.227.30:4761 , piece_id=OQ5P6IQ7GXIT7RJEHCX4IYD4F2AHGRUQ5OW4RMLNAAW76TMFFCFQ, seg_pos={0 0}, upload=475023]: Long tail cancellation due to optimal threshold Dec 28 05:10:01.098687180 (541h41m0.969770235s) [noise=true , address=145.79.227.2:4114 , piece_id=4ELMOELTYFMIUKVFTMNCLB2U5AAGESZJOQ4M54O7KDIWQZWZYVGQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Dec 28 05:10:01.098705855 (541h41m0.9697889s) [noise=true , address=145.79.227.2:4114 , piece_id=4ELMOELTYFMIUKVFTMNCLB2U5AAGESZJOQ4M54O7KDIWQZWZYVGQ, seg_pos={0 0}, upload=475023]: Long tail cancellation due to optimal threshold Dec 28 05:10:01.098952745 (541h41m0.97003579s) [noise=true , address=145.79.227.11:4731 , piece_id=L4QMR4UVQGJ3ME5A3UU3VHDWXGBS5CY6KRG36NRPDZZUUYXXRWAA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Dec 28 05:10:01.098964217 (541h41m0.970047262s) [noise=true , address=145.79.227.11:4731 , piece_id=L4QMR4UVQGJ3ME5A3UU3VHDWXGBS5CY6KRG36NRPDZZUUYXXRWAA, seg_pos={0 0}, upload=475023]: Long tail cancellation due to optimal threshold Dec 28 05:10:01.098992421 (541h41m0.970075456s) [seg_pos={0 0}, upload=475023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:10:01.099101078 (541h41m0.970184133s) [upload=475023]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:10:01.099109974 (541h41m0.97019302s) [upload=475023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:10:01.388984441 (541h41m1.260067486s) [upload=475023]: Done uploading segments... Dec 28 05:10:01.389009329 (541h41m1.260092374s) [upload=475023]: Upload finished. err: <nil> Duration: 6.001737648s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX.] duration: 4.161925s trace: https://jaeger.dev.storj.io/trace/52f4e8411dc99da4 FAIL (2025-12-28 05:11:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.162182169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 42.451578914s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:10:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 42.451613129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [.X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX.XXX] duration: 5.001175885s trace: https://jaeger.dev.storj.io/trace/52f4e8411dc99da4 FAIL (2025-12-28 05:10:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00119958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...X] duration: 30.003509985s trace: https://jaeger.dev.storj.io/trace/85753c584f83639 FAIL (2025-12-28 05:10:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: FOG6ONI3BAUQVLOQ6SPIM5JO2VHV7UNZBS3X6DEJ35MJDN73ED2Q) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: SA55AB5C3PCLPMXQXY3X5IWZMYUKO22O2ETASQKSHJ54I6NXE5XA) context deadline exceeded ======LOGS====== Dec 28 05:09:57.879931513 (541h40m57.751014559s) [upload=475023]: Starting upload Dec 28 05:09:57.880068214 (541h40m57.751151249s) [upload=475023]: Done starting upload Dec 28 05:09:57.880161321 (541h40m57.751244366s) [upload=475023]: Uploading segments... Dec 28 05:09:57.880356253 (541h40m57.751439308s) [upload=475023]: Got next segment. Inline: false Dec 28 05:09:57.880420685 (541h40m57.75150373s) [upload=475023]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:09:57.880427819 (541h40m57.751510854s) [upload=475023]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:09:58.125470970 (541h40m57.996554005s) [seg_pos={0 0}, upload=475023]: Begin upload segment... Dec 28 05:09:58.125530965 (541h40m57.996614s) [seg_pos={0 0}, upload=475023]: Done begin upload segment. Dec 28 05:09:58.125641605 (541h40m57.99672465s) [noise=true , address=145.79.227.18:4006 , piece_id=DNTDZAVTGE7PV5NK26RNVWA7HZB3QBKHD5YCRAETKT7IC7HUSQRA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.126759634 (541h40m57.997842679s) [noise=true , address=145.79.227.31:4798 , piece_id=3CE5D7JHSNWABTEM67E2YNFVHIAR67DRK5YCVKAN7WC6RUTVTYSQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.127371480 (541h40m57.998454515s) [noise=true , address=145.79.227.30:4761 , piece_id=OQ5P6IQ7GXIT7RJEHCX4IYD4F2AHGRUQ5OW4RMLNAAW76TMFFCFQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.127795106 (541h40m57.998878151s) [noise=true , address=145.79.227.41:4085 , piece_id=QCK2FSJR5DPSJMMAQ5IY3JXQHEQMU44ZJ7OWMK4G7V2YTJ3G2YWQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.128191491 (541h40m57.999274536s) [noise=true , address=77.90.31.11:4555 , piece_id=62XIEVBRPTY2N2QIEL5G7P4FK7KMDZFZUF3OZFX6JEU7QVUOHP4A, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.128603285 (541h40m57.99968632s) [noise=true , address=45.91.116.9:4261 , piece_id=JS6QODZKG56DD3QJHLUFUMJY4VJ3GMNZTSLETRE6C5BEBI3ZPCOA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.129014939 (541h40m58.000097974s) [noise=true , address=45.91.116.5:4300 , piece_id=HNG6PXM2AHZNBZALM7DXFVRQQHWNFNWVEIS7HTQO76FSZTTSDM3A, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.129415281 (541h40m58.000498326s) [noise=true , address=145.79.227.10:4403 , piece_id=EZHCRWCAJXOGCXPQIMS2X5QAJFBOA5CQRESBV2OQV6G4I7NIMRZA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.129870668 (541h40m58.000953703s) [noise=true , address=145.79.227.5:4221 , piece_id=XDT6EWP3MY2BCUMIX4XHDS6Z7Q5XC536WRCJPXPIQTBKHE4W3VPQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.130269628 (541h40m58.001352663s) [noise=true , address=45.91.116.6:4974 , piece_id=FOG6ONI3BAUQVLOQ6SPIM5JO2VHV7UNZBS3X6DEJ35MJDN73ED2Q, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.130710467 (541h40m58.001793513s) [noise=true , address=145.79.227.13:4802 , piece_id=YUJFNRRF2U6DACZJD5PD5LSPE4NRVRHRN4OETMOS3BKZ3B62HORQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.131166777 (541h40m58.002249822s) [noise=true , address=145.79.227.16:4909 , piece_id=P4OGPUOTFN25C5MXKJNDUAXJOKUVN3UCHI6B5WYXCNRTU5OCBDRA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.132332706 (541h40m58.003415741s) [noise=true , address=145.79.227.2:4114 , piece_id=4ELMOELTYFMIUKVFTMNCLB2U5AAGESZJOQ4M54O7KDIWQZWZYVGQ, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.132924353 (541h40m58.004007398s) [noise=true , address=145.79.227.11:4731 , piece_id=L4QMR4UVQGJ3ME5A3UU3VHDWXGBS5CY6KRG36NRPDZZUUYXXRWAA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.133431489 (541h40m58.004514534s) [noise=true , address=77.90.31.10:4517 , piece_id=SA55AB5C3PCLPMXQXY3X5IWZMYUKO22O2ETASQKSHJ54I6NXE5XA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:58.134133205 (541h40m58.00521625s) [noise=true , address=145.79.227.15:4873 , piece_id=53G3AIBGMQOWFMXOFIGEH6MWTCSZDNOAYEJWBCQQJLCAEUPRKYEA, seg_pos={0 0}, upload=475023]: Uploading piece... Dec 28 05:09:59.864632552 (541h40m59.735715587s) [upload=475023]: Next returned nil segment Dec 28 05:09:59.864653692 (541h40m59.735736728s) [upload=475023]: Waiting for error group managing segments... Dec 28 05:10:00.154658578 (541h41m0.025741633s) [noise=true , address=145.79.227.5:4221 , piece_id=XDT6EWP3MY2BCUMIX4XHDS6Z7Q5XC536WRCJPXPIQTBKHE4W3VPQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.155028513 (541h41m0.026111548s) [noise=true , address=145.79.227.10:4403 , piece_id=EZHCRWCAJXOGCXPQIMS2X5QAJFBOA5CQRESBV2OQV6G4I7NIMRZA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.160894482 (541h41m0.031977527s) [noise=true , address=145.79.227.15:4873 , piece_id=53G3AIBGMQOWFMXOFIGEH6MWTCSZDNOAYEJWBCQQJLCAEUPRKYEA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.165427544 (541h41m0.036510589s) [noise=true , address=145.79.227.18:4006 , piece_id=DNTDZAVTGE7PV5NK26RNVWA7HZB3QBKHD5YCRAETKT7IC7HUSQRA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.173145761 (541h41m0.044228806s) [noise=true , address=145.79.227.31:4798 , piece_id=3CE5D7JHSNWABTEM67E2YNFVHIAR67DRK5YCVKAN7WC6RUTVTYSQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.185687532 (541h41m0.056770567s) [noise=true , address=145.79.227.13:4802 , piece_id=YUJFNRRF2U6DACZJD5PD5LSPE4NRVRHRN4OETMOS3BKZ3B62HORQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.191748122 (541h41m0.062831167s) [noise=true , address=145.79.227.16:4909 , piece_id=P4OGPUOTFN25C5MXKJNDUAXJOKUVN3UCHI6B5WYXCNRTU5OCBDRA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.322942437 (541h41m0.194025472s) [noise=true , address=77.90.31.11:4555 , piece_id=62XIEVBRPTY2N2QIEL5G7P4FK7KMDZFZUF3OZFX6JEU7QVUOHP4A, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.466260648 (541h41m0.337343693s) [noise=true , address=45.91.116.6:4974 , piece_id=FOG6ONI3BAUQVLOQ6SPIM5JO2VHV7UNZBS3X6DEJ35MJDN73ED2Q, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.496997125 (541h41m0.36808016s) [noise=true , address=45.91.116.5:4300 , piece_id=HNG6PXM2AHZNBZALM7DXFVRQQHWNFNWVEIS7HTQO76FSZTTSDM3A, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.700241796 (541h41m0.571324831s) [noise=true , address=77.90.31.10:4517 , piece_id=SA55AB5C3PCLPMXQXY3X5IWZMYUKO22O2ETASQKSHJ54I6NXE5XA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:00.763585372 (541h41m0.634668407s) [noise=true , address=45.91.116.9:4261 , piece_id=JS6QODZKG56DD3QJHLUFUMJY4VJ3GMNZTSLETRE6C5BEBI3ZPCOA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:01.098168231 (541h41m0.969251276s) [noise=true , address=145.79.227.41:4085 , piece_id=QCK2FSJR5DPSJMMAQ5IY3JXQHEQMU44ZJ7OWMK4G7V2YTJ3G2YWQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: <nil> Dec 28 05:10:01.098193119 (541h41m0.969276164s) [seg_pos={0 0}, upload=475023]: Segment reached optimal threshold of 13 pieces. Dec 28 05:10:01.098437224 (541h41m0.969520279s) [noise=true , address=145.79.227.30:4761 , piece_id=OQ5P6IQ7GXIT7RJEHCX4IYD4F2AHGRUQ5OW4RMLNAAW76TMFFCFQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:10:01.098448094 (541h41m0.969531139s) [noise=true , address=145.79.227.30:4761 , piece_id=OQ5P6IQ7GXIT7RJEHCX4IYD4F2AHGRUQ5OW4RMLNAAW76TMFFCFQ, seg_pos={0 0}, upload=475023]: Long tail cancellation due to optimal threshold Dec 28 05:10:01.098687180 (541h41m0.969770235s) [noise=true , address=145.79.227.2:4114 , piece_id=4ELMOELTYFMIUKVFTMNCLB2U5AAGESZJOQ4M54O7KDIWQZWZYVGQ, seg_pos={0 0}, upload=475023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: expected piece hash; context canceled; EOF Dec 28 05:10:01.098705855 (541h41m0.9697889s) [noise=true , address=145.79.227.2:4114 , piece_id=4ELMOELTYFMIUKVFTMNCLB2U5AAGESZJOQ4M54O7KDIWQZWZYVGQ, seg_pos={0 0}, upload=475023]: Long tail cancellation due to optimal threshold Dec 28 05:10:01.098952745 (541h41m0.97003579s) [noise=true , address=145.79.227.11:4731 , piece_id=L4QMR4UVQGJ3ME5A3UU3VHDWXGBS5CY6KRG36NRPDZZUUYXXRWAA, seg_pos={0 0}, upload=475023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9): protocol: expected piece hash; context canceled; EOF Dec 28 05:10:01.098964217 (541h41m0.970047262s) [noise=true , address=145.79.227.11:4731 , piece_id=L4QMR4UVQGJ3ME5A3UU3VHDWXGBS5CY6KRG36NRPDZZUUYXXRWAA, seg_pos={0 0}, upload=475023]: Long tail cancellation due to optimal threshold Dec 28 05:10:01.098992421 (541h41m0.970075456s) [seg_pos={0 0}, upload=475023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:10:01.099101078 (541h41m0.970184133s) [upload=475023]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:10:01.099109974 (541h41m0.97019302s) [upload=475023]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:10:01.388984441 (541h41m1.260067486s) [upload=475023]: Done uploading segments... Dec 28 05:10:01.389009329 (541h41m1.260092374s) [upload=475023]: Upload finished. err: <nil> Duration: 30.00352922s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX.XX] duration: 5.001573337s trace: https://jaeger.dev.storj.io/trace/7d06dce8658b7959 FAIL (2025-12-28 05:10:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001604948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.692194437s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:10:08 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.692230816s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 47.721308045s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:09:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 47.721330037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX.X] duration: 5.00169193s trace: https://jaeger.dev.storj.io/trace/2656e93265837e66 FAIL (2025-12-28 05:09:51 UTC) orbiter warning: context deadline exceeded Duration: 5.001701438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.04977586s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:09:45 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.049804294s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.065729559s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:09:25 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.065758344s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X...] duration: 11.013495043s trace: https://jaeger.dev.storj.io/trace/34e5fda69dfb1e9 FAIL (2025-12-28 05:09:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e: false Dec 28 05:06:45.280979235 (541h37m45.15206228s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:06:45.280985597 (541h37m45.152068632s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:06:45.526944320 (541h37m45.398027365s) [seg_pos={0 0}, upload=474976]: Begin upload segment... Dec 28 05:06:45.526986921 (541h37m45.398069956s) [seg_pos={0 0}, upload=474976]: Done begin upload segment. Dec 28 05:06:45.527011709 (541h37m45.398094744s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527042748 (541h37m45.398125783s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527515337 (541h37m45.398598382s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527978990 (541h37m45.399062035s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528295442 (541h37m45.399378477s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528624830 (541h37m45.399707865s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528944208 (541h37m45.400027243s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529268124 (541h37m45.400351159s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529637939 (541h37m45.400720974s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531450390 (541h37m45.402533435s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531562423 (541h37m45.402645468s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.532946108 (541h37m45.404029143s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.534501219 (541h37m45.405584254s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.536068473 (541h37m45.407151519s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.537618856 (541h37m45.408701891s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.539358659 (541h37m45.410441704s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:47.145485435 (541h37m47.01656848s) [upload=474976]: Next returned nil segment Dec 28 05:06:47.145501195 (541h37m47.01658425s) [upload=474976]: Waiting for error group managing segments... Dec 28 05:06:47.534273045 (541h37m47.40535609s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.537195640 (541h37m47.408278675s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.547596064 (541h37m47.418679119s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.549122420 (541h37m47.420205455s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567558506 (541h37m47.438641551s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567726545 (541h37m47.43880958s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.568230064 (541h37m47.439313109s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.573777918 (541h37m47.444860963s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.575184647 (541h37m47.446267692s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.685437775 (541h37m47.55652083s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.806684737 (541h37m47.677767782s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.833594478 (541h37m47.704677513s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868044015 (541h37m47.73912706s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868068541 (541h37m47.739151576s) [seg_pos={0 0}, upload=474976]: Segment reached optimal threshold of 13 pieces. Dec 28 05:06:47.868288220 (541h37m47.739371265s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection Dec 28 05:06:47.868298389 (541h37m47.739381434s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868638848 (541h37m47.739721893s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection Dec 28 05:06:47.868648406 (541h37m47.739731441s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868913972 (541h37m47.739997007s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:46536->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:06:47.868924832 (541h37m47.740007877s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868946924 (541h37m47.74002997s) [seg_pos={0 0}, upload=474976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:06:47.869061332 (541h37m47.740144367s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:06:47.869068726 (541h37m47.740151761s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:06:48.169179643 (541h37m48.040262678s) [upload=474976]: Done uploading segments... Dec 28 05:06:48.169198618 (541h37m48.040281653s) [upload=474976]: Upload finished. err: <nil> Duration: 11.013526144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 43.136435423s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:09:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 43.136453337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX.] duration: 5.000939356s trace: https://jaeger.dev.storj.io/trace/6cdc89a9310da29f FAIL (2025-12-28 05:08:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000965295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.391543358s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:08:45 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.391578025s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 43.739839671s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:08:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 43.739867084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X..] duration: 9.385657567s trace: https://jaeger.dev.storj.io/trace/34e5fda69dfb1e9 FAIL (2025-12-28 05:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e: false Dec 28 05:06:45.280979235 (541h37m45.15206228s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:06:45.280985597 (541h37m45.152068632s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:06:45.526944320 (541h37m45.398027365s) [seg_pos={0 0}, upload=474976]: Begin upload segment... Dec 28 05:06:45.526986921 (541h37m45.398069956s) [seg_pos={0 0}, upload=474976]: Done begin upload segment. Dec 28 05:06:45.527011709 (541h37m45.398094744s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527042748 (541h37m45.398125783s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527515337 (541h37m45.398598382s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527978990 (541h37m45.399062035s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528295442 (541h37m45.399378477s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528624830 (541h37m45.399707865s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528944208 (541h37m45.400027243s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529268124 (541h37m45.400351159s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529637939 (541h37m45.400720974s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531450390 (541h37m45.402533435s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531562423 (541h37m45.402645468s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.532946108 (541h37m45.404029143s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.534501219 (541h37m45.405584254s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.536068473 (541h37m45.407151519s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.537618856 (541h37m45.408701891s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.539358659 (541h37m45.410441704s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:47.145485435 (541h37m47.01656848s) [upload=474976]: Next returned nil segment Dec 28 05:06:47.145501195 (541h37m47.01658425s) [upload=474976]: Waiting for error group managing segments... Dec 28 05:06:47.534273045 (541h37m47.40535609s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.537195640 (541h37m47.408278675s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.547596064 (541h37m47.418679119s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.549122420 (541h37m47.420205455s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567558506 (541h37m47.438641551s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567726545 (541h37m47.43880958s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.568230064 (541h37m47.439313109s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.573777918 (541h37m47.444860963s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.575184647 (541h37m47.446267692s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.685437775 (541h37m47.55652083s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.806684737 (541h37m47.677767782s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.833594478 (541h37m47.704677513s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868044015 (541h37m47.73912706s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868068541 (541h37m47.739151576s) [seg_pos={0 0}, upload=474976]: Segment reached optimal threshold of 13 pieces. Dec 28 05:06:47.868288220 (541h37m47.739371265s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection Dec 28 05:06:47.868298389 (541h37m47.739381434s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868638848 (541h37m47.739721893s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection Dec 28 05:06:47.868648406 (541h37m47.739731441s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868913972 (541h37m47.739997007s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:46536->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:06:47.868924832 (541h37m47.740007877s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868946924 (541h37m47.74002997s) [seg_pos={0 0}, upload=474976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:06:47.869061332 (541h37m47.740144367s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:06:47.869068726 (541h37m47.740151761s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:06:48.169179643 (541h37m48.040262678s) [upload=474976]: Done uploading segments... Dec 28 05:06:48.169198618 (541h37m48.040281653s) [upload=474976]: Upload finished. err: <nil> Duration: 9.385691691s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.XX] duration: 5.001410448s trace: https://jaeger.dev.storj.io/trace/6cdc89a9310da29f FAIL (2025-12-28 05:08:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001425067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.767503696s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:07:37 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.7675313s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X.] duration: 6.041046995s trace: https://jaeger.dev.storj.io/trace/34e5fda69dfb1e9 FAIL (2025-12-28 05:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e: false Dec 28 05:06:45.280979235 (541h37m45.15206228s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:06:45.280985597 (541h37m45.152068632s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:06:45.526944320 (541h37m45.398027365s) [seg_pos={0 0}, upload=474976]: Begin upload segment... Dec 28 05:06:45.526986921 (541h37m45.398069956s) [seg_pos={0 0}, upload=474976]: Done begin upload segment. Dec 28 05:06:45.527011709 (541h37m45.398094744s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527042748 (541h37m45.398125783s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527515337 (541h37m45.398598382s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527978990 (541h37m45.399062035s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528295442 (541h37m45.399378477s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528624830 (541h37m45.399707865s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528944208 (541h37m45.400027243s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529268124 (541h37m45.400351159s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529637939 (541h37m45.400720974s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531450390 (541h37m45.402533435s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531562423 (541h37m45.402645468s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.532946108 (541h37m45.404029143s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.534501219 (541h37m45.405584254s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.536068473 (541h37m45.407151519s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.537618856 (541h37m45.408701891s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.539358659 (541h37m45.410441704s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:47.145485435 (541h37m47.01656848s) [upload=474976]: Next returned nil segment Dec 28 05:06:47.145501195 (541h37m47.01658425s) [upload=474976]: Waiting for error group managing segments... Dec 28 05:06:47.534273045 (541h37m47.40535609s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.537195640 (541h37m47.408278675s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.547596064 (541h37m47.418679119s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.549122420 (541h37m47.420205455s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567558506 (541h37m47.438641551s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567726545 (541h37m47.43880958s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.568230064 (541h37m47.439313109s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.573777918 (541h37m47.444860963s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.575184647 (541h37m47.446267692s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.685437775 (541h37m47.55652083s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.806684737 (541h37m47.677767782s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.833594478 (541h37m47.704677513s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868044015 (541h37m47.73912706s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868068541 (541h37m47.739151576s) [seg_pos={0 0}, upload=474976]: Segment reached optimal threshold of 13 pieces. Dec 28 05:06:47.868288220 (541h37m47.739371265s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection Dec 28 05:06:47.868298389 (541h37m47.739381434s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868638848 (541h37m47.739721893s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection Dec 28 05:06:47.868648406 (541h37m47.739731441s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868913972 (541h37m47.739997007s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:46536->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:06:47.868924832 (541h37m47.740007877s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868946924 (541h37m47.74002997s) [seg_pos={0 0}, upload=474976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:06:47.869061332 (541h37m47.740144367s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:06:47.869068726 (541h37m47.740151761s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:06:48.169179643 (541h37m48.040262678s) [upload=474976]: Done uploading segments... Dec 28 05:06:48.169198618 (541h37m48.040281653s) [upload=474976]: Upload finished. err: <nil> Duration: 6.04107634s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 21.366795764s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:07:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 21.366821362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.480923434s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:07:20 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.48094287s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...X] duration: 30.003613537s trace: https://jaeger.dev.storj.io/trace/34e5fda69dfb1e9 FAIL (2025-12-28 05:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e: false Dec 28 05:06:45.280979235 (541h37m45.15206228s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:06:45.280985597 (541h37m45.152068632s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:06:45.526944320 (541h37m45.398027365s) [seg_pos={0 0}, upload=474976]: Begin upload segment... Dec 28 05:06:45.526986921 (541h37m45.398069956s) [seg_pos={0 0}, upload=474976]: Done begin upload segment. Dec 28 05:06:45.527011709 (541h37m45.398094744s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527042748 (541h37m45.398125783s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527515337 (541h37m45.398598382s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.527978990 (541h37m45.399062035s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528295442 (541h37m45.399378477s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528624830 (541h37m45.399707865s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.528944208 (541h37m45.400027243s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529268124 (541h37m45.400351159s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.529637939 (541h37m45.400720974s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531450390 (541h37m45.402533435s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.531562423 (541h37m45.402645468s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.532946108 (541h37m45.404029143s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.534501219 (541h37m45.405584254s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.536068473 (541h37m45.407151519s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.537618856 (541h37m45.408701891s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:45.539358659 (541h37m45.410441704s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Uploading piece... Dec 28 05:06:47.145485435 (541h37m47.01656848s) [upload=474976]: Next returned nil segment Dec 28 05:06:47.145501195 (541h37m47.01658425s) [upload=474976]: Waiting for error group managing segments... Dec 28 05:06:47.534273045 (541h37m47.40535609s) [noise=true , address=145.79.227.15:4875 , piece_id=3P2ID6ZSWD4DTU76LXCAUOW6T4QDZFYLEKAQG7D4OXFVDCXJFY2A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.537195640 (541h37m47.408278675s) [noise=true , address=145.79.227.18:4006 , piece_id=LNU4QEGEULAUA6QF76XRYSJ4TNI7DKABUCK5EWL7PV7BN55PD5KQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.547596064 (541h37m47.418679119s) [noise=true , address=145.79.227.24:4583 , piece_id=GJ35JSY32O7HKOLERDWLUGRLL6BT2CKGDO3LSHM5CRA3WHDVP3BA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.549122420 (541h37m47.420205455s) [noise=true , address=145.79.227.46:4603 , piece_id=5V7XDPCXXSR4KVTCELJ7QB6J3KUOPETZ3DK3CXPZ2743CDQ35DGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567558506 (541h37m47.438641551s) [noise=true , address=145.79.227.2:4114 , piece_id=N6FMEZE52JQLMNZQ2MLCS2ZNP63NN52CS2IQILDIJNE4DBFDQCPQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.567726545 (541h37m47.43880958s) [noise=true , address=145.79.227.14:4840 , piece_id=XULMFCNZ57FRY6VU4MUSVQZF4ESYXC7T7NXLPKHURV5AHNUFPEDA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.568230064 (541h37m47.439313109s) [noise=true , address=145.79.227.38:4977 , piece_id=CLE6JNQXU7OFINORI5LSH4XOL5BWMTPZBBTZDGEUNY6SQVT2NDDQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.573777918 (541h37m47.444860963s) [noise=true , address=145.79.227.41:4085 , piece_id=S3HBDJ6H7G57WCP75XVB4DIW5REJKP2KDEYSK3PXGUQFFZW63ICA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.575184647 (541h37m47.446267692s) [noise=true , address=145.79.227.42:4123 , piece_id=534RPWT72MJWSSEWLWXXLJW6AFGCWFRVIRCJV7FVIG335XAEWWTQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.685437775 (541h37m47.55652083s) [noise=true , address=145.79.227.45:4231 , piece_id=HSKG4IJLLEEJYMUBQVDQHSNFSENEKZRGQHOUSUBZGFWNSCY53BGA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.806684737 (541h37m47.677767782s) [noise=true , address=45.91.116.14:4192 , piece_id=CXPRCHVOB2PNTSUC7H47OPGQ2IMAIF4MGUSSVLUR727QKPMLJC7A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.833594478 (541h37m47.704677513s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3IULPKAIKDW3BFGCB4GMQUBG32GB7XNTMMTOB2A2A5V2D3LQYKA, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868044015 (541h37m47.73912706s) [noise=true , address=77.90.31.11:4556 , piece_id=JAULHDIERA3TBEF3EWGDOMXNBVWH3A2DBDJAOPSRDTQ4QGTFVKYQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: <nil> Dec 28 05:06:47.868068541 (541h37m47.739151576s) [seg_pos={0 0}, upload=474976]: Segment reached optimal threshold of 13 pieces. Dec 28 05:06:47.868288220 (541h37m47.739371265s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46996->45.91.116.5:4297: use of closed network connection Dec 28 05:06:47.868298389 (541h37m47.739381434s) [noise=true , address=45.91.116.5:4297 , piece_id=ROBGBOM6AJ4ZWAQ732JFFP6CGRN4ARZWOKW3Z4QX4WUUUKVUCO3A, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868638848 (541h37m47.739721893s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->45.91.116.9:4262: use of closed network connection Dec 28 05:06:47.868648406 (541h37m47.739731441s) [noise=true , address=45.91.116.9:4262 , piece_id=4SGFS7VUU2FFRUVBEYW3RJVGJGQ72QD7O4JA5MYK6KXARSXXQS6Q, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868913972 (541h37m47.739997007s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:46536->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 05:06:47.868924832 (541h37m47.740007877s) [noise=true , address=45.91.116.11:4119 , piece_id=T6YKD7KSJZPNAADEDRGNZ24P33NVUEYV6YPVJQBGPIJVARPHA6LQ, seg_pos={0 0}, upload=474976]: Long tail cancellation due to optimal threshold Dec 28 05:06:47.868946924 (541h37m47.74002997s) [seg_pos={0 0}, upload=474976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:06:47.869061332 (541h37m47.740144367s) [upload=474976]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:06:47.869068726 (541h37m47.740151761s) [upload=474976]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:06:48.169179643 (541h37m48.040262678s) [upload=474976]: Done uploading segments... Dec 28 05:06:48.169198618 (541h37m48.040281653s) [upload=474976]: Upload finished. err: <nil> Duration: 30.003631602s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.X] duration: 5.001988629s trace: https://jaeger.dev.storj.io/trace/649fa27f796c5d54 FAIL (2025-12-28 05:07:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002010941s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.43654517s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:06:42 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.436573264s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X.] duration: 4.873648378s trace: https://jaeger.dev.storj.io/trace/5326c443cad730cc FAIL (2025-12-28 05:06:36 UTC) orbiter warning: context deadline exceeded Duration: 4.873674988s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 33.515156876s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:06:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 33.515173948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 50.651933681s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:05:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 50.651949831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.197965342s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:05:55 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.197995068s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX...] duration: 5.92053443s trace: https://jaeger.dev.storj.io/trace/aa8c405b9d6f89a FAIL (2025-12-28 05:05:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 05:03:25.083539577 (541h34m24.954622622s) [upload=474930]: Starting upload Dec 28 05:03:25.083626612 (541h34m24.954709647s) [upload=474930]: Done starting upload Dec 28 05:03:25.083696525 (541h34m24.95477957s) [upload=474930]: Uploading segments... Dec 28 05:03:25.083872230 (541h34m24.954955275s) [upload=474930]: Got next segment. Inline: false Dec 28 05:03:25.083929319 (541h34m24.955012364s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:03:25.083936141 (541h34m24.955019176s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:03:25.338498336 (541h34m25.209581371s) [seg_pos={0 0}, upload=474930]: Begin upload segment... Dec 28 05:03:25.338549704 (541h34m25.209632739s) [seg_pos={0 0}, upload=474930]: Done begin upload segment. Dec 28 05:03:25.338577747 (541h34m25.209660792s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.338833044 (541h34m25.209916089s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339414630 (541h34m25.210497666s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339766912 (541h34m25.210849947s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340057866 (541h34m25.211140911s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340407872 (541h34m25.211490907s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340782987 (541h34m25.211866032s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341118505 (541h34m25.21220155s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341480525 (541h34m25.21256356s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342699426 (541h34m25.213782471s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342741726 (541h34m25.213824761s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343815331 (541h34m25.214898376s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343923096 (541h34m25.215006131s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344923541 (541h34m25.216006586s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344997201 (541h34m25.216080236s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.346392789 (541h34m25.217475824s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:26.952520197 (541h34m26.823603242s) [upload=474930]: Next returned nil segment Dec 28 05:03:26.952536228 (541h34m26.823619273s) [upload=474930]: Waiting for error group managing segments... Dec 28 05:03:27.339291868 (541h34m27.210374913s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.339967505 (541h34m27.21105054s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351587261 (541h34m27.222670306s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351742316 (541h34m27.222825351s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352351776 (541h34m27.223434821s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352757619 (541h34m27.223840664s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.357654343 (541h34m27.228737378s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.362524657 (541h34m27.233607702s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.391519668 (541h34m27.262602704s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.586655226 (541h34m27.457738271s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.615530559 (541h34m27.486613594s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.675202583 (541h34m27.546285629s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997707050 (541h34m27.868790095s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997732028 (541h34m27.868815063s) [seg_pos={0 0}, upload=474930]: Segment reached optimal threshold of 13 pieces. Dec 28 05:03:27.997981663 (541h34m27.869064708s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998040785 (541h34m27.86912384s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998200149 (541h34m27.869283194s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998211119 (541h34m27.869294175s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998223263 (541h34m27.869306298s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection Dec 28 05:03:27.998230527 (541h34m27.869313592s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998248771 (541h34m27.869331816s) [seg_pos={0 0}, upload=474930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:03:27.998331740 (541h34m27.869414785s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:03:27.998337981 (541h34m27.869421026s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:03:28.348380875 (541h34m28.21946394s) [upload=474930]: Done uploading segments... Dec 28 05:03:28.348406634 (541h34m28.219489669s) [upload=474930]: Upload finished. err: <nil> Duration: 5.920570089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.X] duration: 5.001133892s trace: https://jaeger.dev.storj.io/trace/5326c443cad730cc FAIL (2025-12-28 05:05:30 UTC) orbiter warning: context deadline exceeded Duration: 5.00115385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.021201713s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:05:19 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.02123622s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 51.098280626s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:05:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 51.098298809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X.] duration: 4.286848616s trace: https://jaeger.dev.storj.io/trace/9f81e4fe0da6893 FAIL (2025-12-28 05:04:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.287031303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX..] duration: 8.257292527s trace: https://jaeger.dev.storj.io/trace/aa8c405b9d6f89a FAIL (2025-12-28 05:04:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 05:03:25.083539577 (541h34m24.954622622s) [upload=474930]: Starting upload Dec 28 05:03:25.083626612 (541h34m24.954709647s) [upload=474930]: Done starting upload Dec 28 05:03:25.083696525 (541h34m24.95477957s) [upload=474930]: Uploading segments... Dec 28 05:03:25.083872230 (541h34m24.954955275s) [upload=474930]: Got next segment. Inline: false Dec 28 05:03:25.083929319 (541h34m24.955012364s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:03:25.083936141 (541h34m24.955019176s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:03:25.338498336 (541h34m25.209581371s) [seg_pos={0 0}, upload=474930]: Begin upload segment... Dec 28 05:03:25.338549704 (541h34m25.209632739s) [seg_pos={0 0}, upload=474930]: Done begin upload segment. Dec 28 05:03:25.338577747 (541h34m25.209660792s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.338833044 (541h34m25.209916089s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339414630 (541h34m25.210497666s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339766912 (541h34m25.210849947s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340057866 (541h34m25.211140911s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340407872 (541h34m25.211490907s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340782987 (541h34m25.211866032s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341118505 (541h34m25.21220155s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341480525 (541h34m25.21256356s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342699426 (541h34m25.213782471s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342741726 (541h34m25.213824761s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343815331 (541h34m25.214898376s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343923096 (541h34m25.215006131s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344923541 (541h34m25.216006586s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344997201 (541h34m25.216080236s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.346392789 (541h34m25.217475824s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:26.952520197 (541h34m26.823603242s) [upload=474930]: Next returned nil segment Dec 28 05:03:26.952536228 (541h34m26.823619273s) [upload=474930]: Waiting for error group managing segments... Dec 28 05:03:27.339291868 (541h34m27.210374913s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.339967505 (541h34m27.21105054s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351587261 (541h34m27.222670306s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351742316 (541h34m27.222825351s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352351776 (541h34m27.223434821s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352757619 (541h34m27.223840664s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.357654343 (541h34m27.228737378s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.362524657 (541h34m27.233607702s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.391519668 (541h34m27.262602704s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.586655226 (541h34m27.457738271s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.615530559 (541h34m27.486613594s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.675202583 (541h34m27.546285629s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997707050 (541h34m27.868790095s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997732028 (541h34m27.868815063s) [seg_pos={0 0}, upload=474930]: Segment reached optimal threshold of 13 pieces. Dec 28 05:03:27.997981663 (541h34m27.869064708s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998040785 (541h34m27.86912384s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998200149 (541h34m27.869283194s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998211119 (541h34m27.869294175s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998223263 (541h34m27.869306298s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection Dec 28 05:03:27.998230527 (541h34m27.869313592s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998248771 (541h34m27.869331816s) [seg_pos={0 0}, upload=474930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:03:27.998331740 (541h34m27.869414785s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:03:27.998337981 (541h34m27.869421026s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:03:28.348380875 (541h34m28.21946394s) [upload=474930]: Done uploading segments... Dec 28 05:03:28.348406634 (541h34m28.219489669s) [upload=474930]: Upload finished. err: <nil> Duration: 8.257322224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X................................................X..........] duration: 40.586798535s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:04:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 40.586860933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.146649172s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:04:13 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.14667948s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX.] duration: 6.175902041s trace: https://jaeger.dev.storj.io/trace/aa8c405b9d6f89a FAIL (2025-12-28 05:04:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 05:03:25.083539577 (541h34m24.954622622s) [upload=474930]: Starting upload Dec 28 05:03:25.083626612 (541h34m24.954709647s) [upload=474930]: Done starting upload Dec 28 05:03:25.083696525 (541h34m24.95477957s) [upload=474930]: Uploading segments... Dec 28 05:03:25.083872230 (541h34m24.954955275s) [upload=474930]: Got next segment. Inline: false Dec 28 05:03:25.083929319 (541h34m24.955012364s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:03:25.083936141 (541h34m24.955019176s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:03:25.338498336 (541h34m25.209581371s) [seg_pos={0 0}, upload=474930]: Begin upload segment... Dec 28 05:03:25.338549704 (541h34m25.209632739s) [seg_pos={0 0}, upload=474930]: Done begin upload segment. Dec 28 05:03:25.338577747 (541h34m25.209660792s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.338833044 (541h34m25.209916089s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339414630 (541h34m25.210497666s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339766912 (541h34m25.210849947s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340057866 (541h34m25.211140911s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340407872 (541h34m25.211490907s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340782987 (541h34m25.211866032s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341118505 (541h34m25.21220155s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341480525 (541h34m25.21256356s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342699426 (541h34m25.213782471s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342741726 (541h34m25.213824761s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343815331 (541h34m25.214898376s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343923096 (541h34m25.215006131s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344923541 (541h34m25.216006586s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344997201 (541h34m25.216080236s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.346392789 (541h34m25.217475824s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:26.952520197 (541h34m26.823603242s) [upload=474930]: Next returned nil segment Dec 28 05:03:26.952536228 (541h34m26.823619273s) [upload=474930]: Waiting for error group managing segments... Dec 28 05:03:27.339291868 (541h34m27.210374913s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.339967505 (541h34m27.21105054s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351587261 (541h34m27.222670306s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351742316 (541h34m27.222825351s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352351776 (541h34m27.223434821s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352757619 (541h34m27.223840664s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.357654343 (541h34m27.228737378s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.362524657 (541h34m27.233607702s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.391519668 (541h34m27.262602704s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.586655226 (541h34m27.457738271s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.615530559 (541h34m27.486613594s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.675202583 (541h34m27.546285629s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997707050 (541h34m27.868790095s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997732028 (541h34m27.868815063s) [seg_pos={0 0}, upload=474930]: Segment reached optimal threshold of 13 pieces. Dec 28 05:03:27.997981663 (541h34m27.869064708s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998040785 (541h34m27.86912384s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998200149 (541h34m27.869283194s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998211119 (541h34m27.869294175s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998223263 (541h34m27.869306298s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection Dec 28 05:03:27.998230527 (541h34m27.869313592s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998248771 (541h34m27.869331816s) [seg_pos={0 0}, upload=474930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:03:27.998331740 (541h34m27.869414785s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:03:27.998337981 (541h34m27.869421026s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:03:28.348380875 (541h34m28.21946394s) [upload=474930]: Done uploading segments... Dec 28 05:03:28.348406634 (541h34m28.219489669s) [upload=474930]: Upload finished. err: <nil> Duration: 6.175917601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX] duration: 30.003996365s trace: https://jaeger.dev.storj.io/trace/aa8c405b9d6f89a FAIL (2025-12-28 05:03:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 05:03:25.083539577 (541h34m24.954622622s) [upload=474930]: Starting upload Dec 28 05:03:25.083626612 (541h34m24.954709647s) [upload=474930]: Done starting upload Dec 28 05:03:25.083696525 (541h34m24.95477957s) [upload=474930]: Uploading segments... Dec 28 05:03:25.083872230 (541h34m24.954955275s) [upload=474930]: Got next segment. Inline: false Dec 28 05:03:25.083929319 (541h34m24.955012364s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:03:25.083936141 (541h34m24.955019176s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:03:25.338498336 (541h34m25.209581371s) [seg_pos={0 0}, upload=474930]: Begin upload segment... Dec 28 05:03:25.338549704 (541h34m25.209632739s) [seg_pos={0 0}, upload=474930]: Done begin upload segment. Dec 28 05:03:25.338577747 (541h34m25.209660792s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.338833044 (541h34m25.209916089s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339414630 (541h34m25.210497666s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339766912 (541h34m25.210849947s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340057866 (541h34m25.211140911s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340407872 (541h34m25.211490907s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340782987 (541h34m25.211866032s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341118505 (541h34m25.21220155s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341480525 (541h34m25.21256356s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342699426 (541h34m25.213782471s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342741726 (541h34m25.213824761s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343815331 (541h34m25.214898376s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343923096 (541h34m25.215006131s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344923541 (541h34m25.216006586s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344997201 (541h34m25.216080236s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.346392789 (541h34m25.217475824s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:26.952520197 (541h34m26.823603242s) [upload=474930]: Next returned nil segment Dec 28 05:03:26.952536228 (541h34m26.823619273s) [upload=474930]: Waiting for error group managing segments... Dec 28 05:03:27.339291868 (541h34m27.210374913s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.339967505 (541h34m27.21105054s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351587261 (541h34m27.222670306s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351742316 (541h34m27.222825351s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352351776 (541h34m27.223434821s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352757619 (541h34m27.223840664s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.357654343 (541h34m27.228737378s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.362524657 (541h34m27.233607702s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.391519668 (541h34m27.262602704s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.586655226 (541h34m27.457738271s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.615530559 (541h34m27.486613594s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.675202583 (541h34m27.546285629s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997707050 (541h34m27.868790095s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997732028 (541h34m27.868815063s) [seg_pos={0 0}, upload=474930]: Segment reached optimal threshold of 13 pieces. Dec 28 05:03:27.997981663 (541h34m27.869064708s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998040785 (541h34m27.86912384s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998200149 (541h34m27.869283194s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998211119 (541h34m27.869294175s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998223263 (541h34m27.869306298s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection Dec 28 05:03:27.998230527 (541h34m27.869313592s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998248771 (541h34m27.869331816s) [seg_pos={0 0}, upload=474930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:03:27.998331740 (541h34m27.869414785s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:03:27.998337981 (541h34m27.869421026s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:03:28.348380875 (541h34m28.21946394s) [upload=474930]: Done uploading segments... Dec 28 05:03:28.348406634 (541h34m28.219489669s) [upload=474930]: Upload finished. err: <nil> Duration: 30.004022165s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X] duration: 5.001978554s trace: https://jaeger.dev.storj.io/trace/9f81e4fe0da6893 FAIL (2025-12-28 05:03:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002004494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.397073303s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:03:40 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.397102938s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................................X.........] duration: 45.522240077s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:03:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 45.522267489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.940246053s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:03:17 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.940285479s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....X] duration: 30.002954931s trace: https://jaeger.dev.storj.io/trace/20d5c6af2296cbf0 FAIL (2025-12-28 05:02:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: YTWAVPDPZKBH4AE2TOUJ7ZQCACAJH2COG3JLZ2VHOKPXEYZGKFYA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 7TKHCNLFATST4VPWMWKZG5MJHQYBSEJ2UOXTWZAWQTVDEIYVT4SA) context deadline exceeded ======LOGS====== 8 05:02:26.828558837 (541h33m26.699641872s) [upload=474914]: Starting upload Dec 28 05:02:26.828622699 (541h33m26.699705734s) [upload=474914]: Done starting upload Dec 28 05:02:26.828825886 (541h33m26.699908931s) [upload=474914]: Uploading segments... Dec 28 05:02:26.828886511 (541h33m26.699969546s) [upload=474914]: Got next segment. Inline: false Dec 28 05:02:26.828918031 (541h33m26.700001066s) [upload=474914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:02:26.828922981 (541h33m26.700006016s) [upload=474914]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:02:27.079426667 (541h33m26.950509712s) [seg_pos={0 0}, upload=474914]: Begin upload segment... Dec 28 05:02:27.079499807 (541h33m26.950582842s) [seg_pos={0 0}, upload=474914]: Done begin upload segment. Dec 28 05:02:27.079538009 (541h33m26.950621044s) [noise=true , address=145.79.227.52:4445 , piece_id=5QCSQ7NN3SL3NMVI4AK4KWJIXCCL5UOSIW64E3JKKH53MOPTMQNQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.079638531 (541h33m26.950721576s) [noise=true , address=145.79.227.7:4293 , piece_id=HYHCYV5IAWJYEMFE57OVAAOPXF3Z6ZZ6RVS5JFSQIXDKZ2DEU2GQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.080279702 (541h33m26.951362747s) [noise=true , address=45.91.116.11:4118 , piece_id=COCMPWPVJ6GJN3HI5SMZ6YGFPI4N5QVE6VJ7MYREFVEECUJMKRUQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.080625741 (541h33m26.951708776s) [noise=true , address=145.79.227.14:4839 , piece_id=NA7CGW3LBYCKUE5DX6HJPDLYWQIMDKMXOKRYOTHX5AQKRDYADYQA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081165969 (541h33m26.952249014s) [noise=true , address=145.79.227.51:4411 , piece_id=SAHB5CSKTHUTEAPJSVKNYCTLAUMUP3N3HBXUZ746KX6X6TMBD4SQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081328389 (541h33m26.952411424s) [noise=true , address=145.79.227.31:4799 , piece_id=5NOMGHHQN3T6YYQ3R2Y7K3BKITIBOSNB5CBEXCEQIYSTHTTZSKRQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081555922 (541h33m26.952638967s) [noise=true , address=145.79.227.25:4617 , piece_id=YFBYT47H7WLRQGYTZXX7GKKZLYSURZMHIM2BUZBZ22W32TZAC52Q, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081856274 (541h33m26.952939309s) [noise=true , address=77.90.31.10:4519 , piece_id=R6WT6MTC7H3IVUO2X3UB5COMENHHN34NAQCEAW2VCDDDQS7U2DWQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082005778 (541h33m26.953088823s) [noise=true , address=145.79.227.9:4365 , piece_id=YO3O23PTGMLNQKSXTR6UNIHTIDMXJ2CNTTWD7YM5I4MKLADZKGEA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082262498 (541h33m26.953345543s) [noise=true , address=145.79.227.12:4767 , piece_id=RT6G4JVCURUQMVQLCQWMBCQRJQ4VJ2WY4W4BUEMXHVIMUXQDHBUQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082496813 (541h33m26.953579858s) [noise=true , address=77.90.31.12:4590 , piece_id=YTWAVPDPZKBH4AE2TOUJ7ZQCACAJH2COG3JLZ2VHOKPXEYZGKFYA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082802195 (541h33m26.95388523s) [noise=true , address=145.79.227.3:4150 , piece_id=PHOQPLSF6K3DHBXB3UWZ54PJCWYLHSEVBQNRKMSBY6HH24GFDDQA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083124458 (541h33m26.954207503s) [noise=true , address=45.91.116.13:4156 , piece_id=7TKHCNLFATST4VPWMWKZG5MJHQYBSEJ2UOXTWZAWQTVDEIYVT4SA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083202047 (541h33m26.954285092s) [noise=true , address=45.91.116.8:4045 , piece_id=TEZFSZWTMK5ONII4QDWSW752F3MOC675NEOW6N3JYB42JYTZZQJA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083570448 (541h33m26.954653493s) [noise=true , address=45.91.116.10:4082 , piece_id=F4ZUIYQ6LB3Z5CYVWENMUUREVHV26JYFGQU6UTBKYMOV7SBJYTFQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083618188 (541h33m26.954701223s) [noise=true , address=145.79.227.17:4971 , piece_id=QLYXCX3VE4H46YM7DFJLQ7WQFRPI7XBPIVL2PZPQPKBZVQPQRL2Q, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:28.719685608 (541h33m28.590768653s) [upload=474914]: Next returned nil segment Dec 28 05:02:28.719706087 (541h33m28.590789132s) [upload=474914]: Waiting for error group managing segments... Dec 28 05:02:29.108045794 (541h33m28.979128829s) [noise=true , address=145.79.227.17:4971 , piece_id=QLYXCX3VE4H46YM7DFJLQ7WQFRPI7XBPIVL2PZPQPKBZVQPQRL2Q, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.109514390 (541h33m28.980597425s) [noise=true , address=145.79.227.14:4839 , piece_id=NA7CGW3LBYCKUE5DX6HJPDLYWQIMDKMXOKRYOTHX5AQKRDYADYQA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.119441714 (541h33m28.990524759s) [noise=true , address=145.79.227.52:4445 , piece_id=5QCSQ7NN3SL3NMVI4AK4KWJIXCCL5UOSIW64E3JKKH53MOPTMQNQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.120192343 (541h33m28.991275378s) [noise=true , address=145.79.227.3:4150 , piece_id=PHOQPLSF6K3DHBXB3UWZ54PJCWYLHSEVBQNRKMSBY6HH24GFDDQA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.120851689 (541h33m28.991934734s) [noise=true , address=145.79.227.7:4293 , piece_id=HYHCYV5IAWJYEMFE57OVAAOPXF3Z6ZZ6RVS5JFSQIXDKZ2DEU2GQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.121343044 (541h33m28.992426079s) [noise=true , address=145.79.227.31:4799 , piece_id=5NOMGHHQN3T6YYQ3R2Y7K3BKITIBOSNB5CBEXCEQIYSTHTTZSKRQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.125167687 (541h33m28.996250722s) [noise=true , address=145.79.227.51:4411 , piece_id=SAHB5CSKTHUTEAPJSVKNYCTLAUMUP3N3HBXUZ746KX6X6TMBD4SQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.145068240 (541h33m29.016151275s) [noise=true , address=145.79.227.25:4617 , piece_id=YFBYT47H7WLRQGYTZXX7GKKZLYSURZMHIM2BUZBZ22W32TZAC52Q, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.154369251 (541h33m29.025452286s) [noise=true , address=145.79.227.12:4767 , piece_id=RT6G4JVCURUQMVQLCQWMBCQRJQ4VJ2WY4W4BUEMXHVIMUXQDHBUQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.242443788 (541h33m29.113526843s) [noise=true , address=145.79.227.9:4365 , piece_id=YO3O23PTGMLNQKSXTR6UNIHTIDMXJ2CNTTWD7YM5I4MKLADZKGEA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.356424034 (541h33m29.227507069s) [noise=true , address=77.90.31.12:4590 , piece_id=YTWAVPDPZKBH4AE2TOUJ7ZQCACAJH2COG3JLZ2VHOKPXEYZGKFYA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.474876244 (541h33m29.34595928s) [noise=true , address=45.91.116.13:4156 , piece_id=7TKHCNLFATST4VPWMWKZG5MJHQYBSEJ2UOXTWZAWQTVDEIYVT4SA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.640545333 (541h33m29.511628368s) [noise=true , address=45.91.116.11:4118 , piece_id=COCMPWPVJ6GJN3HI5SMZ6YGFPI4N5QVE6VJ7MYREFVEECUJMKRUQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.640563889 (541h33m29.511646924s) [seg_pos={0 0}, upload=474914]: Segment reached optimal threshold of 13 pieces. Dec 28 05:02:29.640723252 (541h33m29.511806287s) [noise=true , address=77.90.31.10:4519 , piece_id=R6WT6MTC7H3IVUO2X3UB5COMENHHN34NAQCEAW2VCDDDQS7U2DWQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:60168->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:60168->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60168->77.90.31.10:4519: use of closed network connection Dec 28 05:02:29.640729453 (541h33m29.511812498s) [noise=true , address=77.90.31.10:4519 , piece_id=R6WT6MTC7H3IVUO2X3UB5COMENHHN34NAQCEAW2VCDDDQS7U2DWQ, seg_pos={0 0}, upload=474914]: Long tail cancellation due to optimal threshold Dec 28 05:02:29.640938251 (541h33m29.512021296s) [noise=true , address=45.91.116.10:4082 , piece_id=F4ZUIYQ6LB3Z5CYVWENMUUREVHV26JYFGQU6UTBKYMOV7SBJYTFQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 05:02:29.640949353 (541h33m29.512032398s) [noise=true , address=45.91.116.10:4082 , piece_id=F4ZUIYQ6LB3Z5CYVWENMUUREVHV26JYFGQU6UTBKYMOV7SBJYTFQ, seg_pos={0 0}, upload=474914]: Long tail cancellation due to optimal threshold Dec 28 05:02:29.641046196 (541h33m29.512129231s) [noise=true , address=45.91.116.8:4045 , piece_id=TEZFSZWTMK5ONII4QDWSW752F3MOC675NEOW6N3JYB42JYTZZQJA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 05:02:29.641050966 (541h33m29.512134001s) [noise=true , address=45.91.116.8:4045 , piece_id=TEZFSZWTMK5ONII4QDWSW752F3MOC675NEOW6N3JYB42JYTZZQJA, seg_pos={0 0}, upload=474914]: Long tail cancellation due to optimal threshold Dec 28 05:02:29.641063941 (541h33m29.512146986s) [seg_pos={0 0}, upload=474914]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:02:29.641150074 (541h33m29.512233109s) [upload=474914]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:02:29.641154634 (541h33m29.512237679s) [upload=474914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:02:29.947730710 (541h33m29.818813766s) [upload=474914]: Done uploading segments... Dec 28 05:02:29.947756619 (541h33m29.818839674s) [upload=474914]: Upload finished. err: <nil> Duration: 30.003003854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................................X........] duration: 28.835529334s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:02:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 28.835559751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..] duration: 4.36848358s trace: https://jaeger.dev.storj.io/trace/7d10056725899236 FAIL (2025-12-28 05:02:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.368510171s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)