Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.898015829s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:36:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.898047892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................XX.......] duration: 5.969360262s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:35:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.969393595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................XX......] duration: 5.765582899s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:34:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.765611763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 5.780718042s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:33:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780817512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 5.727729878s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:32:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.727749364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 5.637005962s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:31:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.637045788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 5.696885873s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:30:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.696914508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 5.913114789s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:29:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.913145427s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.245452738s trace: https://jaeger.dev.storj.io/trace/55ff491f49891b49 FAIL (2025-11-25 16:29:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 16:28:47.321164002 (392h24m45.872634453s) [upload=345561]: Starting upload Nov 25 16:28:47.321244084 (392h24m45.872714535s) [upload=345561]: Done starting upload Nov 25 16:28:47.321342281 (392h24m45.872812742s) [upload=345561]: Uploading segments... Nov 25 16:28:47.321480945 (392h24m45.872951396s) [upload=345561]: Got next segment. Inline: false Nov 25 16:28:47.321521653 (392h24m45.872992114s) [upload=345561]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 16:28:47.321528445 (392h24m45.872998897s) [upload=345561]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 16:28:47.569226625 (392h24m46.120697086s) [seg_pos={0 0}, upload=345561]: Begin upload segment... Nov 25 16:28:47.569277603 (392h24m46.120748054s) [seg_pos={0 0}, upload=345561]: Done begin upload segment. Nov 25 16:28:47.569301709 (392h24m46.12077217s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569345382 (392h24m46.120815833s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.569941547 (392h24m46.121412008s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570293928 (392h24m46.121764379s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570788309 (392h24m46.12225877s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.570913107 (392h24m46.122383578s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.571412828 (392h24m46.122883279s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572270512 (392h24m46.123740973s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.572772387 (392h24m46.124242838s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.573920553 (392h24m46.125391014s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574282523 (392h24m46.125752984s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.574774430 (392h24m46.126244881s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.575325579 (392h24m46.12679603s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576162292 (392h24m46.127632774s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.576973507 (392h24m46.128443968s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:47.578218949 (392h24m46.12968941s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Uploading piece... Nov 25 16:28:49.407103784 (392h24m47.958574245s) [upload=345561]: Next returned nil segment Nov 25 16:28:49.407229494 (392h24m47.958699955s) [upload=345561]: Waiting for error group managing segments... Nov 25 16:28:49.883090241 (392h24m48.434560692s) [noise=true , address=77.90.31.12:4590 , piece_id=QFHXJWSCVA6OTXJMMATFJY3F3IC2F73VMRSBEUAFIGW4H4ASDEKQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.082752298 (392h24m48.634222749s) [noise=true , address=145.79.227.9:4366 , piece_id=MO3KDHLLSO4Z3J4GP7WUE7RPTBBVTBNQG735B3JJVPDFFIBLJXDA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:50.125258639 (392h24m48.676729101s) [noise=true , address=145.79.227.28:4727 , piece_id=D7CPKVTZS5IXOVUTSFNGRTACTEW2CJXAEGNWHJKJ7QFPKPLZFJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:51.076449024 (392h24m49.627919476s) [noise=true , address=145.79.227.34:4906 , piece_id=OPT3Y2OOSZVCXKOIFM5DS52X5DWFAPNITHRDRMYHMFCTWIT5WYKA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:53.354542190 (392h24m51.906012641s) [noise=true , address=145.79.227.14:4837 , piece_id=7WMZSQAWRJ5NLWJW4P6OGN66F3ULITOPPGFIJDN3KV5DOVZZHGFA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:28:55.609075859 (392h24m54.160546321s) [noise=true , address=145.79.227.40:4051 , piece_id=YL7ZITSRXBQS74QPNLOAITBGZDDSYOWKA4FHC6CBBZ7GSVKKLCCQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:05.493697001 (392h25m4.045167462s) [noise=true , address=145.79.227.19:4078 , piece_id=NE32VKDRLOJZM4NFT5WEZLJNYC52XTBHCNUZAQJNWBN2GASWWXYQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:08.267247120 (392h25m6.818717581s) [noise=true , address=145.79.227.42:4123 , piece_id=3ULFN7YVKFNAOKXXEPYYDCDWRPI2HF6A6HEMK5ZI4JMP6UEFF2TA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:10.168294941 (392h25m8.719765402s) [noise=true , address=145.79.227.23:4547 , piece_id=AAEFH4MGHLRDMRFHQ22R3PDY32FHTQRGKLDSZHKPBXH5Y476QTHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:11.208620898 (392h25m9.760091359s) [noise=true , address=145.79.227.25:4618 , piece_id=VY4I2FFCMPQ7C63OPYDMAMIU577API5E5CKKVHHSND4WTK2ADB5A, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:14.440065197 (392h25m12.991535668s) [noise=true , address=145.79.227.24:4582 , piece_id=CEWTJCCUNLWWTFP3WD2PK75AJSFAN65ADXTNVNHYFGF7MRATZRUA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:15.559910707 (392h25m14.111381168s) [noise=true , address=145.79.227.18:4006 , piece_id=DILD5MCP2LWRNKMASK3JR4WNWB7HQ5SUYTBLGQ24A26JA4X7RBNA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: <nil> Nov 25 16:29:16.627776183 (392h25m15.179246644s) [noise=true , address=145.79.227.13:4801 , piece_id=VNW643IY5QZV7CCSE5KLJBXES3R3PTAILRDVTOX6TQPF2WQ7JJRQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34574->145.79.227.13:4801: use of closed network connection Nov 25 16:29:16.627908315 (392h25m15.179378776s) [noise=true , address=145.79.227.32:4836 , piece_id=ZMO6MBJ4PARVLE4K4JQZLVIZF2NADGWKYHCPCXMBPZQ6A234IENQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41258->145.79.227.32:4836: use of closed network connection Nov 25 16:29:16.627972317 (392h25m15.179442778s) [noise=true , address=145.79.227.10:4402 , piece_id=SKZIJE7F3H4V2LAMR6IBXHHTV4ZDKTIB2FDMMNJL4F25BHNYSKHA, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55454->145.79.227.10:4402: use of closed network connection Nov 25 16:29:16.628074692 (392h25m15.179545153s) [noise=true , address=145.79.227.6:4257 , piece_id=2SEYCRNQ3HS4BQLFRB4Y6T2GUZIPWFNZUDMAVKNTBYVXXROKBURQ, seg_pos={0 0}, upload=345561]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43084->145.79.227.6:4257: use of closed network connection Nov 25 16:29:16.628111291 (392h25m15.179581752s) [seg_pos={0 0}, upload=345561]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 16:29:16.870655766 (392h25m15.422126227s) [upload=345561]: Done uploading segments... Nov 25 16:29:16.870701794 (392h25m15.422172265s) [upload=345561]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.245463948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.248575529s trace: https://jaeger.dev.storj.io/trace/74ad0651c9ca0337 FAIL (2025-11-25 16:28:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 674142886s) [seg_pos={0 0}, upload=345554]: Done begin upload segment. Nov 25 16:28:17.122702962 (392h24m15.674173423s) [noise=true , address=145.79.227.31:4798 , piece_id=URCWIJ6UKZURG2PM5R525JETAOPNZVZW66TAK6OFXP3EN6O5XHPA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.122753869 (392h24m15.674224331s) [noise=true , address=145.79.227.13:4803 , piece_id=SUOGA2NVIDJLEOYVFVXDUAS7ABOZJZIN6NBOTCSF2NUNZIEFP2FQ, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.123575444 (392h24m15.675045905s) [noise=true , address=145.79.227.28:4726 , piece_id=APCT6AVFRFR4CINAZ6LR77UR3Z7YJKQ5FNMC3QKPNJMKJ6U3O23Q, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.123962411 (392h24m15.675432862s) [noise=true , address=145.79.227.42:4123 , piece_id=EN55DFNLHRVVDPAY6EXQOGY5LWKQMQN56SZDHSSJRD54AKRKDBHA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.124325572 (392h24m15.675796033s) [noise=true , address=145.79.227.41:4086 , piece_id=EIM2RANQX7PDNKSI5UBEFW5VKLGWF4LCVMQJDAYFD2ZQVV7BRRAA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.124672773 (392h24m15.676143234s) [noise=true , address=77.90.31.19:4841 , piece_id=MAUVOPGMGENYDM5U7EIPDYUSCT47TVKT6WCA72LPV4ENGTSK7WVQ, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.125192894 (392h24m15.676663355s) [noise=true , address=145.79.227.34:4906 , piece_id=ZELIWMFNJJZI6NTOUIOZ6EVFF5LBCBPVQC4HNWNVKUOWDGEHBYCA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.125553361 (392h24m15.677023812s) [noise=true , address=145.79.227.30:4763 , piece_id=KZTOFQ74ECZXNHOX7FBCENRJBRT64KER4GJMPEE73BCUJ4YBJXSQ, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.127370521 (392h24m15.678840982s) [noise=true , address=145.79.227.33:4871 , piece_id=5D7BZXPWEAF75OBRTTXC2XUF2LQH5PAFY6VUVM65YFVHPPK46ILA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.127834093 (392h24m15.679304554s) [noise=true , address=145.79.227.26:4654 , piece_id=BAW6ZPMWQBEMGCME6LAVDTBFBSXGYPD5KJ5BM2WCH7EMO7FNYYTA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.129486039 (392h24m15.6809565s) [noise=true , address=145.79.227.25:4618 , piece_id=4BVWOWH6UY37X22TW73LV5IY2B3F6AYKEUNNPRAFNKND46NRYBUA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.131085545 (392h24m15.682556006s) [noise=true , address=145.79.227.8:4331 , piece_id=TT3URNFCKOY3BOHJLWUJA5LS7HFELB22VDMSPW4TBYXEQWMGIN6A, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.132453099 (392h24m15.68392356s) [noise=true , address=145.79.227.19:4078 , piece_id=U6BNYWG5JBHZ2EZNXS65YBIPVFSQIY2TG45DRHDUEC7TPQJDJBDQ, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.134165489 (392h24m15.68563595s) [noise=true , address=145.79.227.21:4473 , piece_id=TXEEPRL65AR3JT2PXYJ2MDENC37Y7UIESAGUSJ3HCB5JO47EYFOQ, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.135644085 (392h24m15.687114546s) [noise=true , address=145.79.227.17:4970 , piece_id=SOGN77NNXQ3PVYP32AGJXU7BKXA6KIVCKBLSCEPMET7ORYPMH6WA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:17.136553266 (392h24m15.688023728s) [noise=true , address=145.79.227.24:4581 , piece_id=WDWIXUVRIRFHHDOL2R37B2P2ISVCKQ62INZYESVSX3Q3PTK27YGA, seg_pos={0 0}, upload=345554]: Uploading piece... Nov 25 16:28:18.868599133 (392h24m17.420069584s) [upload=345554]: Next returned nil segment Nov 25 16:28:18.868615002 (392h24m17.420085463s) [upload=345554]: Waiting for error group managing segments... Nov 25 16:28:19.147537659 (392h24m17.69900812s) [noise=true , address=145.79.227.26:4654 , piece_id=BAW6ZPMWQBEMGCME6LAVDTBFBSXGYPD5KJ5BM2WCH7EMO7FNYYTA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:19.149500928 (392h24m17.700971389s) [noise=true , address=145.79.227.24:4581 , piece_id=WDWIXUVRIRFHHDOL2R37B2P2ISVCKQ62INZYESVSX3Q3PTK27YGA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:19.219062578 (392h24m17.770533039s) [noise=true , address=145.79.227.19:4078 , piece_id=U6BNYWG5JBHZ2EZNXS65YBIPVFSQIY2TG45DRHDUEC7TPQJDJBDQ, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:19.281403176 (392h24m17.832873637s) [noise=true , address=77.90.31.19:4841 , piece_id=MAUVOPGMGENYDM5U7EIPDYUSCT47TVKT6WCA72LPV4ENGTSK7WVQ, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:25.268225140 (392h24m23.819695591s) [noise=true , address=145.79.227.31:4798 , piece_id=URCWIJ6UKZURG2PM5R525JETAOPNZVZW66TAK6OFXP3EN6O5XHPA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:29.967548726 (392h24m28.519019187s) [noise=true , address=145.79.227.33:4871 , piece_id=5D7BZXPWEAF75OBRTTXC2XUF2LQH5PAFY6VUVM65YFVHPPK46ILA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:30.893589791 (392h24m29.445060252s) [noise=true , address=145.79.227.21:4473 , piece_id=TXEEPRL65AR3JT2PXYJ2MDENC37Y7UIESAGUSJ3HCB5JO47EYFOQ, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:34.154255678 (392h24m32.705726139s) [noise=true , address=145.79.227.28:4726 , piece_id=APCT6AVFRFR4CINAZ6LR77UR3Z7YJKQ5FNMC3QKPNJMKJ6U3O23Q, seg_pos={0 0}, upload=345554]: Done uploading piece. err: <nil> Nov 25 16:28:46.161345107 (392h24m44.712815579s) [noise=true , address=145.79.227.41:4086 , piece_id=EIM2RANQX7PDNKSI5UBEFW5VKLGWF4LCVMQJDAYFD2ZQVV7BRRAA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: write tcp 188.166.225.170:45876->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.41:4086: use of closed network connection Nov 25 16:28:46.161624359 (392h24m44.71309481s) [noise=true , address=145.79.227.34:4906 , piece_id=ZELIWMFNJJZI6NTOUIOZ6EVFF5LBCBPVQC4HNWNVKUOWDGEHBYCA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: write tcp 188.166.225.170:40422->145.79.227.34:4906: use of closed network connection; write tcp 188.166.225.170:40422->145.79.227.34:4906: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40422->145.79.227.34:4906: use of closed network connection Nov 25 16:28:46.162088644 (392h24m44.713559105s) [noise=true , address=145.79.227.42:4123 , piece_id=EN55DFNLHRVVDPAY6EXQOGY5LWKQMQN56SZDHSSJRD54AKRKDBHA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: read tcp 188.166.225.170:52724->145.79.227.42:4123: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 16:28:46.162824395 (392h24m44.714294856s) [noise=true , address=145.79.227.30:4763 , piece_id=KZTOFQ74ECZXNHOX7FBCENRJBRT64KER4GJMPEE73BCUJ4YBJXSQ, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:50018->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 16:28:46.163555627 (392h24m44.715026088s) [noise=true , address=145.79.227.13:4803 , piece_id=SUOGA2NVIDJLEOYVFVXDUAS7ABOZJZIN6NBOTCSF2NUNZIEFP2FQ, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:145.79.227.13:4803): protocol: rpc: dial tcp 145.79.227.13:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 16:28:46.163826472 (392h24m44.715296933s) [noise=true , address=145.79.227.17:4970 , piece_id=SOGN77NNXQ3PVYP32AGJXU7BKXA6KIVCKBLSCEPMET7ORYPMH6WA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:145.79.227.17:4970): protocol: read tcp 188.166.225.170:39808->145.79.227.17:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 16:28:46.164024850 (392h24m44.715495301s) [noise=true , address=145.79.227.8:4331 , piece_id=TT3URNFCKOY3BOHJLWUJA5LS7HFELB22VDMSPW4TBYXEQWMGIN6A, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:33978->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:33978->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33978->145.79.227.8:4331: use of closed network connection Nov 25 16:28:46.164166670 (392h24m44.715637201s) [noise=true , address=145.79.227.25:4618 , piece_id=4BVWOWH6UY37X22TW73LV5IY2B3F6AYKEUNNPRAFNKND46NRYBUA, seg_pos={0 0}, upload=345554]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: read tcp 188.166.225.170:38612->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 16:28:46.164207147 (392h24m44.715677598s) [seg_pos={0 0}, upload=345554]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 25 16:28:46.408168951 (392h24m44.959639422s) [upload=345554]: Done uploading segments... Nov 25 16:28:46.408236760 (392h24m44.959707221s) [upload=345554]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.248601229s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.378372681s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 15:00:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.378404312s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.566195019s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 15:00:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566228073s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.577606178s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:59:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.577629231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.525764602s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:58:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525793877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.444950536s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:57:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444975324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.583974875s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:57:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584010432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.487424589s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:56:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487453173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.453286998s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:55:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.453343274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 4.407447631s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:55:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.407473892s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.521460875s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:55:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.521515079s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.552238562s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:55:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552265984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.997640445s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:54:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.997673508s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.556615794s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:53:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.556645621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.819455911s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:52:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.819483924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.50119627s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:52:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.50122786s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.735498122s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:51:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.735528819s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.499174088s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:50:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499200657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.614898543s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:49:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.614954258s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.589912462s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:49:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589937921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.65470914s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:47:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654739438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.55770204s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:47:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.557732638s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.548829864s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:46:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548862697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 4.008979417s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:45:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.009008631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.531393129s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:44:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531429557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.584287328s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:44:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584497238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.484905748s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:43:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484932318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 4.457795274s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:42:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.457827988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.554007762s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:41:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554035665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.601725105s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:41:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.601754543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 4.131369529s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:40:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.131406768s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.401612238s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:39:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.401677302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.612454576s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:38:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.612497787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.44080586s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:37:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440870984s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.576643223s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:36:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.576759112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.476165362s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:35:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.47622666s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.984243608s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:34:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.984273486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.645718829s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:33:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.645747625s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.63393926s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:33:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.633968445s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.672083942s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:32:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.672112185s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.613136486s trace: https://jaeger.dev.storj.io/trace/51ca7972bc9c9e55 FAIL (2025-11-25 14:31:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613168006s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)