Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 5.831101205s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:12:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.831131323s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 697.065494ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:12:55 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 697.096194ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 5.853093818s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.853125849s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 693.643849ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:11:48 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 693.681529ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 6.079744479s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:11:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.079770778s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 233.659705ms target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/290d15fae21b1e85 FAIL (2025-11-25 17:10:41 UTC) orbiter warning: Get "https://us1.storj.io/login": dial tcp 35.212.10.183:443: connect: connection refused Duration: 233.680054ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 5.913387569s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:10:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.91341367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 5.991614793s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:09:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.991633127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 6.009127012s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:08:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.00924203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 5.790041133s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:07:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.790070668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 5.799247523s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:06:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.799281738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 6.044518029s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:05:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.044551122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 6.004592844s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:05:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.004647619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 6.011151731s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.011184261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 6.025855965s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:04:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.025917152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 6.271109877s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:04:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.271140696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 6.136665341s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:03:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.136830324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 6.015913208s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:03:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.015961219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 5.935066892s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:02:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.935095677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 6.164270281s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:01:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.164299958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 6.081916924s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 17:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.082019871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 6.126925811s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:59:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.126957272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 6.203102284s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:58:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.203133633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 5.810271262s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:57:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.810381291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 5.928284638s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:56:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.928315136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 6.084299457s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:55:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.084329583s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 6.203470767s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:55:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.203774635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 5.926369991s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:54:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.926384538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 6.160097963s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:53:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.160128362s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 6.426006757s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:52:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.426034542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 5.683176618s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:51:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.683212314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 5.919303892s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:50:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.91932339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 5.825580993s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:49:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82564226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 5.821502991s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.821520985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 5.70703163s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:48:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.707075874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 5.704628578s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:47:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.704672332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.702415229s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.702444104s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 5.777346247s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:46:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.777379009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 5.72176143s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:45:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.721775518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.793435591s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:44:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793467011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 5.922572308s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:43:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.92260485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 5.943850204s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:43:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.943881363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 5.833784476s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:42:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83381207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 5.86088454s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:41:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.860913906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 5.708494122s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.708524549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 5.694191261s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:39:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.694220327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.753372187s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:39:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.753402153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 5.894356239s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.894374523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 5.847084913s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:38:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.847112156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 5.869557827s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:37:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.869582202s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)