Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X............X...............] duration: 21.521265169s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:40:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 21.521299174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X............X..............] duration: 5.787114362s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:39:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.78714479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X............X.............] duration: 5.939271239s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:38:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.939300276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X............X............] duration: 5.899521888s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:37:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.899579658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X............X...........] duration: 5.760160181s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:36:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.76019234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X............X..........] duration: 5.675490977s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:35:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.675522788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X............X.........] duration: 5.685870511s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:35:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.685904848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X............X........] duration: 5.718460721s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:34:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.718492322s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.688093638s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:34:09 UTC) orbiter warning: context deadline exceeded Duration: 5.688122053s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X............X.......] duration: 5.875463516s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:33:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.875494375s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.642907351s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:32:55 UTC) orbiter warning: context deadline exceeded Duration: 5.642930876s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X............X......] duration: 5.736488732s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:32:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.736517687s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.785670785s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:32:10 UTC) orbiter warning: context deadline exceeded Duration: 5.785702235s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.934504984s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:31:23 UTC) orbiter warning: context deadline exceeded Duration: 5.934535363s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X............X.....] duration: 5.726687495s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:31:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.726754623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X............X....] duration: 5.700818506s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:30:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.700847032s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.650038764s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:30:29 UTC) orbiter warning: context deadline exceeded Duration: 5.650066357s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X............X...] duration: 7.025260328s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:30:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.025294673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X............X..] duration: 5.870372787s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:29:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.870393406s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.214778422s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:29:15 UTC) orbiter warning: context deadline exceeded Duration: 5.214808949s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X............X.] duration: 9.163845599s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:28:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.163864356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X............X] duration: 30.251970911s trace: https://jaeger.dev.storj.io/trace/7edb5354aa71e52b FAIL (2025-11-27 20:28:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 20:27:57.302814805 (444h23m55.854285266s) [upload=391335]: Got next segment. Inline: false Nov 27 20:27:57.302855252 (444h23m55.854325713s) [upload=391335]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:27:57.302862085 (444h23m55.854332547s) [upload=391335]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:27:57.550049281 (444h23m56.101519762s) [seg_pos={0 0}, upload=391335]: Begin upload segment... Nov 27 20:27:57.550140093 (444h23m56.101610554s) [seg_pos={0 0}, upload=391335]: Done begin upload segment. Nov 27 20:27:57.550208343 (444h23m56.101678804s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.550517953 (444h23m56.101988414s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551618268 (444h23m56.103088729s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.551669275 (444h23m56.103139736s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552081700 (444h23m56.103552161s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552356884 (444h23m56.103827346s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.552662036 (444h23m56.104132497s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553021189 (444h23m56.10449165s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553310931 (444h23m56.104781392s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.553744327 (444h23m56.105214788s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554170688 (444h23m56.105641149s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554507860 (444h23m56.105978322s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.554766583 (444h23m56.106237044s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.555237921 (444h23m56.106708392s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556128035 (444h23m56.107598496s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:57.556896809 (444h23m56.10836726s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Uploading piece... Nov 27 20:27:59.291903037 (444h23m57.843373508s) [upload=391335]: Next returned nil segment Nov 27 20:27:59.291957922 (444h23m57.843428413s) [upload=391335]: Waiting for error group managing segments... Nov 27 20:27:59.615556550 (444h23m58.167027011s) [noise=true , address=77.90.31.15:4698 , piece_id=ENGYNYV6JTHRKA2W3JVW5I5HJ5M4TFUZNPG3U3N3UBFJPLDMYAPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.627368122 (444h23m58.178838583s) [noise=true , address=77.90.31.17:4771 , piece_id=Y66K72PYBNMBZZD7DWFD324UWSUKU6BSUHS5FZHJT3T5OFUCS47A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.635043267 (444h23m58.186513728s) [noise=true , address=77.90.31.11:4553 , piece_id=AAM3WFEQ3MFREEWS2FQ5CNVOXA4AOSGI6DDTSA7DP7EO6RU3EKZQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.637304242 (444h23m58.188774693s) [noise=true , address=45.91.116.4:4938 , piece_id=J7SI2GC6TJJDFRPCPFFT22KIF6AKWL2YNXH2QRVOLJJUB3HIFZ5A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:27:59.670338115 (444h23m58.221808577s) [noise=true , address=77.90.31.14:4661 , piece_id=OIM7VVXRNUACXSLWGSJ4EPMH6OLLAWLIYWB7PMQWII3XUEHAURZA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.144187571 (444h23m58.695658043s) [noise=true , address=145.79.227.9:4367 , piece_id=GWNENIPQHK22B6TMXMJWSXYF3EDWWAJSEFOJVNQN5FPOYWY6BFPA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.281690746 (444h23m58.833161207s) [noise=true , address=145.79.227.13:4804 , piece_id=OT7GCB66SVZNYFLG6QMXOKTHZOURX4C7J55JHCLC5LSYQASXHTLA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.322916307 (444h23m58.874386759s) [noise=true , address=145.79.227.38:4979 , piece_id=QNB73D5GAGM7CG67PO2YZXUOJHHDNC6QLS6FS6USGL26DILMBFTQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.514807969 (444h23m59.06627844s) [noise=true , address=145.79.227.30:4761 , piece_id=RDI2IMWTD7AI26RFP5EYATGHHGMFSV4RMYBBGNV5RHF5RMTHDH6A, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:00.926249869 (444h23m59.47772032s) [noise=true , address=145.79.227.34:4907 , piece_id=EIXTOAOTV5CWBS4QVXMMF7E7ZUYCKBR4B52IJJF63RP6EZ6OX6CQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:01.005428080 (444h23m59.556898541s) [noise=true , address=145.79.227.32:4836 , piece_id=XC5P4CKJTKT3KALUJ2WVNX4UQSCGGMDY3GQM22N5PGEYJQHIOEAA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: <nil> Nov 27 20:28:26.611275652 (444h24m25.162746113s) [noise=true , address=145.79.227.10:4401 , piece_id=Y3JYXQK5U2FBDGUW5PTEMCRK2GNL3NKPHKUPIDWEQLMA6EH24SGA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45404->145.79.227.10:4401: use of closed network connection Nov 27 20:28:26.611469500 (444h24m25.162939961s) [noise=true , address=145.79.227.17:4971 , piece_id=ANICBT3SC5PD7Q6QLYD4Y2TYSSNFTI5CBA2IRTSOLD3VX47FGEQA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57244->145.79.227.17:4971: use of closed network connection Nov 27 20:28:26.611614025 (444h24m25.163084476s) [noise=true , address=145.79.227.42:4121 , piece_id=6NGLQ6CMIJMPCGZDSD6226GW2MGMEQELH4E557SPGHZFKHUG55CA, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40200->145.79.227.42:4121: use of closed network connection Nov 27 20:28:26.611707714 (444h24m25.163178175s) [noise=true , address=145.79.227.8:4330 , piece_id=IBVBSEX4Y4MCYE4Y2K7DVETAUY77PD6H2JGCCJFMKQPGLF4ZBGOQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:125pLUE8mHgK5ZuhphNhDfjNB2kTM88jKNtraf7MMUDC9DU2YVW, address:145.79.227.8:4330): protocol: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37058->145.79.227.8:4330: use of closed network connection Nov 27 20:28:26.611791313 (444h24m25.163261774s) [noise=true , address=145.79.227.14:4840 , piece_id=IHAKLTST7PL3LII3OMN2XPTEJAKIWZURVCIWPZU4YKX2BVUMQMMQ, seg_pos={0 0}, upload=391335]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37490->145.79.227.14:4840: use of closed network connection Nov 27 20:28:26.611815850 (444h24m25.163286301s) [seg_pos={0 0}, upload=391335]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:28:26.861568591 (444h24m25.413039052s) [upload=391335]: Done uploading segments... Nov 27 20:28:26.861610971 (444h24m25.413081432s) [upload=391335]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.251998273s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.971831472s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:28:26 UTC) orbiter warning: context deadline exceeded Duration: 5.971863903s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.701343118s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:28:06 UTC) orbiter warning: context deadline exceeded Duration: 5.701366082s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.74759066s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:27:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.747616199s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.692177109s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:27:05 UTC) orbiter warning: context deadline exceeded Duration: 5.692208117s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.762163996s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:27:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.762194634s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.282504097s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:25:58 UTC) orbiter warning: context deadline exceeded Duration: 6.282580312s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.906717773s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:25:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.906748361s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.810548457s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:25:41 UTC) orbiter warning: context deadline exceeded Duration: 5.810585608s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.020388606s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.020416497s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.846604937s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:24:26 UTC) orbiter warning: context deadline exceeded Duration: 5.846632581s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.712721083s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.712748786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.796501869s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:23:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.796529242s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.261921115s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:23:19 UTC) orbiter warning: context deadline exceeded Duration: 6.261948597s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 11.248518941s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.24855045s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.120118822s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:22:52 UTC) orbiter warning: context deadline exceeded Duration: 6.120235936s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.975048964s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:22:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.975075715s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.026167362s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:21:58 UTC) orbiter warning: context deadline exceeded Duration: 6.026367462s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.966652069s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:21:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.966669482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.747843663s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:21:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.747868902s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.323328938s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:21:15 UTC) orbiter warning: context deadline exceeded Duration: 6.323361399s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.931104659s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:20:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.931137192s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.210111039s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:20:16 UTC) orbiter warning: context deadline exceeded Duration: 6.210137098s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 9.06851926s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:19:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.068538197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.245858404s trace: https://jaeger.dev.storj.io/trace/2060b99581203f14 FAIL (2025-11-27 20:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 27 20:19:03.667726375 (444h15m2.219196836s) [upload=391208]: Starting upload Nov 27 20:19:03.667790167 (444h15m2.219260628s) [upload=391208]: Done starting upload Nov 27 20:19:03.668017700 (444h15m2.219488161s) [upload=391208]: Uploading segments... Nov 27 20:19:03.668059440 (444h15m2.219529901s) [upload=391208]: Got next segment. Inline: false Nov 27 20:19:03.668090799 (444h15m2.219561251s) [upload=391208]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 27 20:19:03.668096039 (444h15m2.2195665s) [upload=391208]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 27 20:19:03.916503848 (444h15m2.467974339s) [seg_pos={0 0}, upload=391208]: Begin upload segment... Nov 27 20:19:03.916564143 (444h15m2.468034604s) [seg_pos={0 0}, upload=391208]: Done begin upload segment. Nov 27 20:19:03.916606383 (444h15m2.468076854s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.916673100 (444h15m2.468143561s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917442936 (444h15m2.468913397s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.917847406 (444h15m2.469317868s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918306571 (444h15m2.469777032s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.918880153 (444h15m2.470350614s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919316704 (444h15m2.470787165s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919667843 (444h15m2.471138304s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.919954139 (444h15m2.4714246s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920330434 (444h15m2.471800895s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920600619 (444h15m2.47207107s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.920756476 (444h15m2.472226937s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921035027 (444h15m2.472505488s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921477428 (444h15m2.472947889s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.921624980 (444h15m2.473095441s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:03.922228198 (444h15m2.473698649s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Uploading piece... Nov 27 20:19:05.681385997 (444h15m4.232856458s) [upload=391208]: Next returned nil segment Nov 27 20:19:05.681432656 (444h15m4.232903107s) [upload=391208]: Waiting for error group managing segments... Nov 27 20:19:05.962712590 (444h15m4.514183051s) [noise=true , address=77.90.31.19:4841 , piece_id=2RZFRSYBSJBRUEW5735QFLV223ZJJQJPNVBIT7KFVI6IELJYQ74A, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.014906027 (444h15m4.566376488s) [noise=true , address=77.90.31.15:4699 , piece_id=AVBW7PJM4OJHECHL7ZMO2S6R32X3MGOCTA5EUECOLRUASQHFHVRA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.055803946 (444h15m4.607274407s) [noise=true , address=77.90.31.21:4879 , piece_id=VR2B7MVEEVIGTYOUSOGZNJAPL7AIIDSNOQOVCOT6KYXTDXMGTEFA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:06.084272055 (444h15m4.635742517s) [noise=true , address=77.90.31.17:4772 , piece_id=OXU5OY5NNJ2NMEC45ZVQFX7BYGERCGMN7NOGSK2GUTWO2TXEBT5Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.071674622 (444h15m5.623145084s) [noise=true , address=145.79.227.16:4909 , piece_id=7Y7EA2XABUMDSOF3AXH5SEXVYPWNW7BA5JRDKK3COWW3EHDOJYWA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.509248613 (444h15m6.060719074s) [noise=true , address=145.79.227.30:4761 , piece_id=4GLCO7CALVNDYPP5ZTL6EITIDE6VSOEJO5S7UR2XHTVJIUBLNN7Q, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.517448137 (444h15m6.068918588s) [noise=true , address=145.79.227.34:4906 , piece_id=S4ZQZ7RCRQTBOFSTHHBV4LNHRQVG5GFJGPVIHZ6WF7VS2MTYKCZQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.573861559 (444h15m6.125332031s) [noise=true , address=145.79.227.14:4838 , piece_id=46BJFVN6KIK5YT6TGMPLKHQDMSGA3NUQ6NCTNU7VJ2NLFLQLYZYQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.587848724 (444h15m6.139319175s) [noise=true , address=145.79.227.22:4509 , piece_id=UKN4WJBAODLI2QDUX5BQIRNYGTHJKWAFWFT7QZBYK2N2LMRZT7SA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:07.985637726 (444h15m6.537108177s) [noise=true , address=145.79.227.4:4187 , piece_id=SDP5OQBIE65X76XCA2OVUNHSUQ4HQFWC4QJU7HEJ4BMHZN6DEMVQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:10.264295549 (444h15m8.81576601s) [noise=true , address=145.79.227.23:4547 , piece_id=LOTI2HQPSMRU6I6GW63ANFJO4OSTPOMQJWVZ4E252RHKWMGXDHLA, seg_pos={0 0}, upload=391208]: Done uploading piece. err: <nil> Nov 27 20:19:32.971663657 (444h15m31.523134108s) [noise=true , address=145.79.227.42:4122 , piece_id=4QFNLUGFU2QI2NEMRFVVSUPHZZH4CU4JCRZGGOXWGBP2QWOVIJTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59162->145.79.227.42:4122: use of closed network connection Nov 27 20:19:32.971774919 (444h15m31.52324537s) [noise=true , address=145.79.227.40:4050 , piece_id=D3OFY6OAVWDYOV4WVHMFUWN4MSWXJXF4EV7JHBKRTL3SD243AXTQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38590->145.79.227.40:4050: use of closed network connection Nov 27 20:19:32.971821437 (444h15m31.523291908s) [noise=true , address=145.79.227.20:4438 , piece_id=QFPOXCVLWCM2EYDZBNYBDFT6NETMN4UBMEDEC5V5YJEAGIPSGAMQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49386->145.79.227.20:4438: use of closed network connection Nov 27 20:19:32.972134614 (444h15m31.523605075s) [noise=true , address=145.79.227.25:4619 , piece_id=NHFQ3FY4CPRN6HOLY62UZA3USRZCRLJBTHM7YWXXFPCYZKFTBUIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: read tcp 188.166.225.170:35810->145.79.227.25:4619: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972180410 (444h15m31.523650872s) [noise=true , address=145.79.227.11:4731 , piece_id=JKGLMMVLIIWH455A3NGALLIDEQOBXC52RPE4LWEIJXTDBQR3OQIQ, seg_pos={0 0}, upload=391208]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: read tcp 188.166.225.170:57302->145.79.227.11:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 27 20:19:32.972223944 (444h15m31.523694405s) [seg_pos={0 0}, upload=391208]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 27 20:19:33.215914512 (444h15m31.767384963s) [upload=391208]: Done uploading segments... Nov 27 20:19:33.215976739 (444h15m31.7674472s) [upload=391208]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.245885105s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.966302664s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:19:20 UTC) orbiter warning: context deadline exceeded Duration: 5.966336196s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.890640214s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:18:12 UTC) orbiter warning: context deadline exceeded Duration: 5.890733139s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.701878836s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:17:07 UTC) orbiter warning: context deadline exceeded Duration: 5.701937828s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.824672794s trace: https://jaeger.dev.storj.io/trace/539555662bc9d577 FAIL (2025-11-27 20:16:31 UTC) orbiter warning: context deadline exceeded Duration: 5.824702198s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)