Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.516857929s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516915899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.69309781s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:07:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.693127865s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 5.001111973s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:06:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001135036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.815224614s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:06:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.815256455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.844771617s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:05:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.844802195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.442280157s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:05:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442310556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.408229467s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:04:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.408250037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.756529399s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:04:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.756559718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.620603219s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:03:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.62064605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.782900442s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:03:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.782928647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.759453636s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:02:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.759481399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.498844005s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:02:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.498870126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.721047872s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:02:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.72107864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.558759309s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:01:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558821057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.820250706s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:01:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.820277659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.382321374s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:01:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.382407999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.793235383s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.793358226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.73629619s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 23:00:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.736325897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.48364729s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 23:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483681064s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.735025471s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:59:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.735056399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.510090073s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:59:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510112757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.79262971s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:59:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.792648976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.777644484s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:59:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.777659924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.521947078s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:58:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.521975152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.78527884s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:58:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.78534677s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.35680726s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:57:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.356835625s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.753335896s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:57:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.753365273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.478293515s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:56:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.478330866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.531759453s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:56:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531786925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.791011028s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:56:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.791042467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.720828474s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:55:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.720877737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.470477163s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:55:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470503903s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.588811377s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:55:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.58884008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.551406329s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:55:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.551435886s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.720580522s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:54:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.720605198s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.456315605s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:54:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456342968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.731507228s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.731526676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.500424798s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:53:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500451438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.534040212s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:52:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.534065962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.391103969s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:52:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.39113135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.823164979s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:52:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.823189207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.441344451s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:52:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441371592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.803160494s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:51:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.803186735s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.412948414s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:51:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.412973282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.741930545s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:51:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.741956664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.791400698s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:50:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.79142877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.882725371s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:50:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.882755368s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.434389477s trace: https://jaeger.dev.storj.io/trace/65363ea90eb05175 FAIL (2025-11-28 22:50:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.434423812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.785537767s trace: https://jaeger.dev.storj.io/trace/216ceff4333b156a FAIL (2025-11-28 22:50:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem_ObjectBegin Nov 28 22:31:07.354394010 (470h27m5.905864471s) [upload=414260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:07.602274485 (470h27m6.153744946s) [seg_pos={0 0}, upload=414260]: Begin upload segment... Nov 28 22:31:07.602366480 (470h27m6.153836941s) [seg_pos={0 0}, upload=414260]: Done begin upload segment. Nov 28 22:31:07.602430182 (470h27m6.153900643s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.604358603 (470h27m6.155829064s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.605777355 (470h27m6.157247816s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.606714209 (470h27m6.15818467s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.607495887 (470h27m6.158966358s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608209547 (470h27m6.159679998s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.608880233 (470h27m6.160350695s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.609635342 (470h27m6.161105803s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.610250092 (470h27m6.161720553s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.611041960 (470h27m6.162512421s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.615308464 (470h27m6.166778915s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616011393 (470h27m6.167481844s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.616670758 (470h27m6.168141219s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.617171652 (470h27m6.168642113s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.620719247 (470h27m6.172189698s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621159806 (470h27m6.172630267s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.621560489 (470h27m6.17303095s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:07.622035804 (470h27m6.173506255s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Uploading piece... Nov 28 22:31:08.920650996 (470h27m7.472121447s) [upload=414260]: Next returned nil segment Nov 28 22:31:08.920667977 (470h27m7.472138438s) [upload=414260]: Waiting for error group managing segments... Nov 28 22:31:09.251358015 (470h27m7.802828476s) [noise=true , address=64.49.13.137:4498 , piece_id=QWZEGEOYJ6AKIKVWGXMLNUSRUSXAQKL25BQR5OYMJBKS44U75Q6Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.333280618 (470h27m7.884751069s) [noise=true , address=64.49.13.137:4496 , piece_id=V6SQ5S2PCEGB7KKMLCNIHCASDSKLR4HARSLU6YCKVJGVUNNWCZOA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.489027585 (470h27m8.040498036s) [noise=true , address=145.79.227.18:4016 , piece_id=RBLIHWR6DB56PHDGDX2BKOD2FVMYIUUDMOATKTAW7TI6HQKVW4YA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.495349323 (470h27m8.046819784s) [noise=true , address=145.79.227.10:4413 , piece_id=MEHNYJFEZNCKU2NOYZQZ55I457XDV3OLPNEDVB5HTZJSM7EDRYNQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.500059121 (470h27m8.051529582s) [noise=true , address=145.79.227.22:4525 , piece_id=X7K2GSTRLMYHF3ZIWSENZ77RLUWUVAZS3RXUFJ52DBSESQ5OH3JQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.502887166 (470h27m8.054357627s) [noise=true , address=145.79.227.52:4462 , piece_id=2RQOU34KUMNL76GJXLGAPOJ33PVBORKNKHTJFX5VNPACAUL7E63A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.503657774 (470h27m8.055128235s) [noise=true , address=145.79.227.28:4731 , piece_id=YYNCOROOPX7P2HVRLZ6AT26KZJ43ZOYRMT64NWEEFJHBMICOIS2A, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.506427478 (470h27m8.057897939s) [noise=true , address=145.79.227.16:4919 , piece_id=ZUPQ7UGYLRPZBNZYHCLCLK4GALLSHKUSO2MVLWANV7FJNUYMD2QA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.513298772 (470h27m8.064769233s) [noise=true , address=145.79.227.5:4254 , piece_id=7FANMVUJLKH3P6ZV7UT2RSMVPIIVGYA5SXVGINHEBM3Y55B7N7IA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.520746183 (470h27m8.072216644s) [noise=true , address=77.90.31.19:4871 , piece_id=EWMC7HMJ67CEJVLL3TPOD756YRHWSKPRRXGUQI6U5JJSJC7JWA4Q, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.527410172 (470h27m8.078880633s) [noise=true , address=77.90.31.21:4888 , piece_id=BDXW46K772UDQQTWRJJJRBJBIHUUVIXNPACDHBW77JFCAAPDMGAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:31:09.613410601 (470h27m8.164881052s) [noise=true , address=77.90.31.12:4606 , piece_id=KCBSTA4PIR2C3ICQJE4QXFP7NCG637EH7TNXRT2FNIEJFOIPHKOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: <nil> Nov 28 22:32:06.651559230 (470h28m5.203029691s) [noise=true , address=45.91.116.9:4269 , piece_id=HEAOYCEYG73XL7HTKOGSSAB22X3GJG4U4YCO52POHNCYPPFDZFXA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: read tcp 188.166.225.170:54708->45.91.116.9:4269: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651606320 (470h28m5.203076781s) [noise=true , address=45.91.116.10:4108 , piece_id=YZDADFTPSWPTMN2PZXV7PRK6WOWBHWJRKHBTSAM6MRZ336FPWBAQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:124taxpXdSnzEkq6UG1ZPi7cKLkaoDSc3G9FH1YHbGbDwMTM8PG, address:45.91.116.10:4108): protocol: rpc: dial tcp 45.91.116.10:4108: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Nov 28 22:32:06.651768629 (470h28m5.20323908s) [noise=true , address=45.91.116.13:4170 , piece_id=7XW4EUR5PTFKO2CIAIZZJKAW44WZ6PYI5A5LZSYBWCEOY3YPQEOQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1acQG8MjLvgzmCs8ELswcsKJrNvYmGQLX1FL4dHeCj2AUH7A5R, address:45.91.116.13:4170): protocol: read tcp 188.166.225.170:55806->45.91.116.13:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.651863830 (470h28m5.203334292s) [noise=true , address=45.91.116.8:4057 , piece_id=FABI2X2W7QXI3BYLZH7ASZT5KGXBCQ2NWMFKWD2VXOY5PTV2FCUQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1S5wNu2NCY5ad2m2684ZFjaXeM68KMC4YpkvTUgbpSMrzJjj5x, address:45.91.116.8:4057): protocol: rpc: dial tcp 45.91.116.8:4057: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652034606 (470h28m5.203505057s) [noise=true , address=45.91.116.7:4033 , piece_id=XAFMHMC3MNS3VVKQWN2MVJIJLYFQVWAVTK7XI7IGYXAT5FVRCLTQ, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1Mg5mxUjW56quDidSgA788D7ByfAzvsEuWJqwtzY4bLyxRn3mu, address:45.91.116.7:4033): protocol: read tcp 188.166.225.170:49814->45.91.116.7:4033: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:06.652306322 (470h28m5.203776783s) [noise=true , address=45.91.116.4:4971 , piece_id=DJLP5EAP43D6GVRBNFMOB6XCPJTM7WTGLBGT3QSRCF5CM6VMTTBA, seg_pos={0 0}, upload=414260]: Done uploading piece. err: ecclient: upload failed (node:1mAbf2K1TfQwCpEET5ndPZxmHxP7NaLDPPWowiy9Yrvzv4QvMV, address:45.91.116.4:4971): protocol: context deadline exceeded; context deadline exceeded Nov 28 22:32:06.652950359 (470h28m5.20442082s) [seg_pos={0 0}, upload=414260]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 28 22:32:06.909602786 (470h28m5.461073257s) [upload=414260]: Done uploading segments... Nov 28 22:32:06.909644555 (470h28m5.461115016s) [upload=414260]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 5.785557184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.932349765s trace: https://jaeger.dev.storj.io/trace/1300d4029b0c8c71 FAIL (2025-11-28 22:50:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.932385593s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)