Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 33.256839136s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:51:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 33.256869985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X..............XXXX............] duration: 10.288636883s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:51:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.288664987s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.520400313s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:51:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.520440939s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X.....X...X.......X.........X.............X........] duration: 4.804501627s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:51:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.804526627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X..............XXXX...........] duration: 8.934188461s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:50:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.934218167s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [................X...X...X..XX.........XXX................X..] duration: 8.958485739s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:50:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.958768999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 27.194641926s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:50:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 27.194673075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X..............XXXX..........] duration: 12.819205207s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:50:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.819253659s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.170430232s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:50:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.170454769s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.................X...X...X..XX.........XXX................X.] duration: 11.045468931s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:50:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.045486665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X.....X...X.......X.........X.............X.......] duration: 3.803842669s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:50:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.803873127s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..................X...X...X..XX.........XXX................X] duration: 1m0.001831385s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:49:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001852836s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 28.717656434s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:49:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 28.717684157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.332880527s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:49:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.332909803s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X..............XXXX.........] duration: 8.897397018s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.897425582s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.154228289s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:49:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.154330713s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 22.933105854s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:48:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 22.933136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X..............XXXX........] duration: 8.744678244s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:48:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.744748317s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X..........X.....X...X.......X.........X.............X......] duration: 3.459839027s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:48:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.459859716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.780114953s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:48:46 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.780143209s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...................X...X...X..XX.........XXX................] duration: 9.944086233s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:48:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.944183539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.391096904s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:48:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.391125529s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X..........X.....X...X.......X.........X.............X.....] duration: 4.731364151s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:48:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.731398075s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 21.148276315s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:48:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 21.148310459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.382354152s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:47:53 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.382383838s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X..............XXXX.......] duration: 8.876920658s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:47:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.876991333s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....................X...X...X..XX.........XXX...............] duration: 6.624854046s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:47:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.62488159s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X..........X.....X...X.......X.........X.............X....] duration: 4.971356801s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:47:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.971507096s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....................X...X...X..XX.........XXX..............] duration: 9.02412258s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:47:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.024145063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001488965s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:47:06 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 30.001584416s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................................X..............XXXX......] duration: 12.000196031s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:47:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 12.000224776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 17.719898808s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:46:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 17.719929485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X..........X.....X...X.......X.........X.............X...] duration: 5.000848994s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:46:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000871857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......................X...X...X..XX.........XXX.............] duration: 8.540308922s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:46:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.540365511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................................X..............XXXX.....] duration: 22.670915213s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:45:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 22.671006326s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X..........X.....X...X.......X.........X.............X..] duration: 4.654900717s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:45:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.654915265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 16.704998607s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:45:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.705028133s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......................X...X...X..XX.........XXX............] duration: 8.508831197s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:45:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.50885918s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X..............XXXX....] duration: 8.951376837s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:45:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.951408026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........................X...X...X..XX.........XXX...........] duration: 8.570532842s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:45:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.570589741s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X..........X.....X...X.......X.........X.............X.] duration: 5.00172489s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:45:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001754686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 21.075903759s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:44:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 21.075923627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 31.089891489s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:44:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 31.089922529s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........................X...X...X..XX.........XXX..........] duration: 8.948748484s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:44:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.948781867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................X..............XXXX...] duration: 8.891895107s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.891924193s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.....X..........X.....X...X.......X.........X.............X] duration: 5.001957623s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:44:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001986398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........................X...X...X..XX.........XXX.........] duration: 9.7900587s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:43:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.790090411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................X..............XXXX..] duration: 19.545486056s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:43:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 19.545525902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 18.889135674s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:43:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.889168287s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........................X...X...X..XX.........XXX........] duration: 10.81961164s trace: https://jaeger.dev.storj.io/trace/6569213f1db796e9 FAIL (2025-12-09 12:43:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.819639693s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)