Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.530066377s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:27:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530094951s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.383994901s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:27:29 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.384022213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.] duration: 17.564178354s trace: https://jaeger.dev.storj.io/trace/3a1fd42ac932b285 FAIL (2025-12-29 18:27:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ) context deadline exceeded ======LOGS====== 18:26:33.813705818 (578h57m33.684788863s) [upload=507532]: Starting upload Dec 29 18:26:33.813874650 (578h57m33.684957705s) [upload=507532]: Done starting upload Dec 29 18:26:33.813958349 (578h57m33.685041394s) [upload=507532]: Uploading segments... Dec 29 18:26:33.814412383 (578h57m33.685495418s) [upload=507532]: Got next segment. Inline: false Dec 29 18:26:33.814457921 (578h57m33.685540966s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:26:33.814471216 (578h57m33.685554261s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:26:34.063810996 (578h57m33.934894041s) [seg_pos={0 0}, upload=507532]: Begin upload segment... Dec 29 18:26:34.063864398 (578h57m33.934947433s) [seg_pos={0 0}, upload=507532]: Done begin upload segment. Dec 29 18:26:34.063896880 (578h57m33.934979925s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.063974087 (578h57m33.935057122s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064553570 (578h57m33.935636606s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064878950 (578h57m33.935961995s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065159133 (578h57m33.936242168s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065533095 (578h57m33.936616131s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066004624 (578h57m33.937087659s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066309954 (578h57m33.93739299s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066589918 (578h57m33.937672953s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068018888 (578h57m33.939101924s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068812069 (578h57m33.939895114s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.070383621 (578h57m33.941466656s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.071867497 (578h57m33.942950532s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.073144719 (578h57m33.944227754s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.074638453 (578h57m33.945721488s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.076525787 (578h57m33.947608832s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:35.761702838 (578h57m35.632785873s) [upload=507532]: Next returned nil segment Dec 29 18:26:35.761724058 (578h57m35.632807113s) [upload=507532]: Waiting for error group managing segments... Dec 29 18:26:36.033563302 (578h57m35.904646347s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.052393928 (578h57m35.923476963s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.057903458 (578h57m35.928986503s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.060886178 (578h57m35.931969213s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.068023248 (578h57m35.939106294s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.069710270 (578h57m35.940793315s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.075557525 (578h57m35.94664056s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.086714258 (578h57m35.957797294s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.087295215 (578h57m35.95837825s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.094639470 (578h57m35.965722505s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.233550958 (578h57m36.104633993s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.256068244 (578h57m36.127151279s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322162147 (578h57m36.193245182s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322185030 (578h57m36.193268065s) [seg_pos={0 0}, upload=507532]: Segment reached optimal threshold of 13 pieces. Dec 29 18:26:36.322387746 (578h57m36.193470791s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322396553 (578h57m36.193479588s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322401503 (578h57m36.193484538s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection Dec 29 18:26:36.322410490 (578h57m36.193493525s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322545437 (578h57m36.193628472s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322553001 (578h57m36.193636046s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322574341 (578h57m36.193657386s) [seg_pos={0 0}, upload=507532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:26:36.322812254 (578h57m36.19389529s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:26:36.322820801 (578h57m36.193903847s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:26:36.616226831 (578h57m36.487309866s) [upload=507532]: Done uploading segments... Dec 29 18:26:36.616245626 (578h57m36.487328661s) [upload=507532]: Upload finished. err: <nil> Duration: 17.564264097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX] duration: 30.002428129s trace: https://jaeger.dev.storj.io/trace/3a1fd42ac932b285 FAIL (2025-12-29 18:27:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ) context deadline exceeded ======LOGS====== 18:26:33.813705818 (578h57m33.684788863s) [upload=507532]: Starting upload Dec 29 18:26:33.813874650 (578h57m33.684957705s) [upload=507532]: Done starting upload Dec 29 18:26:33.813958349 (578h57m33.685041394s) [upload=507532]: Uploading segments... Dec 29 18:26:33.814412383 (578h57m33.685495418s) [upload=507532]: Got next segment. Inline: false Dec 29 18:26:33.814457921 (578h57m33.685540966s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:26:33.814471216 (578h57m33.685554261s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:26:34.063810996 (578h57m33.934894041s) [seg_pos={0 0}, upload=507532]: Begin upload segment... Dec 29 18:26:34.063864398 (578h57m33.934947433s) [seg_pos={0 0}, upload=507532]: Done begin upload segment. Dec 29 18:26:34.063896880 (578h57m33.934979925s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.063974087 (578h57m33.935057122s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064553570 (578h57m33.935636606s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.064878950 (578h57m33.935961995s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065159133 (578h57m33.936242168s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.065533095 (578h57m33.936616131s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066004624 (578h57m33.937087659s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066309954 (578h57m33.93739299s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.066589918 (578h57m33.937672953s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068018888 (578h57m33.939101924s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.068812069 (578h57m33.939895114s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.070383621 (578h57m33.941466656s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.071867497 (578h57m33.942950532s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.073144719 (578h57m33.944227754s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.074638453 (578h57m33.945721488s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:34.076525787 (578h57m33.947608832s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Uploading piece... Dec 29 18:26:35.761702838 (578h57m35.632785873s) [upload=507532]: Next returned nil segment Dec 29 18:26:35.761724058 (578h57m35.632807113s) [upload=507532]: Waiting for error group managing segments... Dec 29 18:26:36.033563302 (578h57m35.904646347s) [noise=true , address=145.79.227.11:4729 , piece_id=PZ52MD27J7SPXJQLCW7RJ4P2XECZZOTFTJVP64TX543M6FB2CYBA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.052393928 (578h57m35.923476963s) [noise=true , address=145.79.227.24:4581 , piece_id=NAM6O6TWOGWTHSLW4T62I2KZHBZVXN3HDURV7CNJI3MHFV6G3XJA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.057903458 (578h57m35.928986503s) [noise=true , address=145.79.227.4:4185 , piece_id=XDMVQTKV5JCIAVZMC4PT2YQPM5EXO4YO3N75QHUGAAHUJK76RH5Q, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.060886178 (578h57m35.931969213s) [noise=true , address=145.79.227.2:4115 , piece_id=J3VBXGNF4YC4EZYOJCKTKCDSWTB6PTUQMN5NDPB26HXALTR5SRDQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.068023248 (578h57m35.939106294s) [noise=true , address=145.79.227.8:4329 , piece_id=PJAES7N5AJF5ZYOLBI5WFKKNPHBGK5NL4KJF7EMDMK4QMQTQJDTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.069710270 (578h57m35.940793315s) [noise=true , address=145.79.227.32:4835 , piece_id=ODK3ZQEKNL5Z4TBZXC2YZMME4NGGBPQRDO2U2V7ZF5EKG56F5PNA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.075557525 (578h57m35.94664056s) [noise=true , address=145.79.227.15:4874 , piece_id=JLM2YQDE5Y337HVT7V2RO7CXX6EE74TTRSS5SPOIPVBAQVYIKWAA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.086714258 (578h57m35.957797294s) [noise=true , address=145.79.227.26:4655 , piece_id=JRLU3VWUGKGAAUT3WVXWH75BUYJKMXMZ7MCKSWKANV5ZTEQV7FEA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.087295215 (578h57m35.95837825s) [noise=true , address=145.79.227.39:4015 , piece_id=VHNJ57KTDBQDMNWOCTSNITXNRDYGWPSG4U47UKDZ2RDT77LXO7OA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.094639470 (578h57m35.965722505s) [noise=true , address=145.79.227.52:4445 , piece_id=JE6VULINM2UHOSPP5POHIPYCKNXK7HO2U3XEVCGVH3ZNYPTMCB7A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.233550958 (578h57m36.104633993s) [noise=true , address=77.90.31.14:4662 , piece_id=4DWEBBGFSBIVG4MTIM6HGLPPVJMRTODFPWTVSIO7HJZJTFXAT2AA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.256068244 (578h57m36.127151279s) [noise=true , address=77.90.31.12:4590 , piece_id=R4OWZJTOAVKLUSSKN6PVH64LBZYD5AECMX6KNQEOX42CZ25LPP6A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322162147 (578h57m36.193245182s) [noise=true , address=45.91.116.5:4300 , piece_id=OYHRDD6CBS3HHHY5Q2QOAKUXD7SDEZHON5WIVS6IIIDQLEMM3YTQ, seg_pos={0 0}, upload=507532]: Done uploading piece. err: <nil> Dec 29 18:26:36.322185030 (578h57m36.193268065s) [seg_pos={0 0}, upload=507532]: Segment reached optimal threshold of 13 pieces. Dec 29 18:26:36.322387746 (578h57m36.193470791s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322396553 (578h57m36.193479588s) [noise=true , address=45.91.116.7:4010 , piece_id=5EAHIRLEJHGPFI5M3VBGE33H437YCYJQ4Y47UQU7OCJ3KOVJ4FDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322401503 (578h57m36.193484538s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59776->145.79.227.41:4086: use of closed network connection Dec 29 18:26:36.322410490 (578h57m36.193493525s) [noise=true , address=145.79.227.41:4086 , piece_id=7UZSYMYEHFEZGULD5L7O5UPXWQZQ3P6D53WAFNEAJYU6SCDMKX3A, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322545437 (578h57m36.193628472s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: expected piece hash; context canceled; EOF Dec 29 18:26:36.322553001 (578h57m36.193636046s) [noise=true , address=145.79.227.51:4411 , piece_id=4IKHN4M7Q65OOU4MUTU5XJBNZC2DVCBEOOPA4S4KGJW2DOPSFXDA, seg_pos={0 0}, upload=507532]: Long tail cancellation due to optimal threshold Dec 29 18:26:36.322574341 (578h57m36.193657386s) [seg_pos={0 0}, upload=507532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:26:36.322812254 (578h57m36.19389529s) [upload=507532]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:26:36.322820801 (578h57m36.193903847s) [upload=507532]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:26:36.616226831 (578h57m36.487309866s) [upload=507532]: Done uploading segments... Dec 29 18:26:36.616245626 (578h57m36.487328661s) [upload=507532]: Upload finished. err: <nil> Duration: 30.00244491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.460620759s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:26:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460644524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 35.539971698s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:26:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.539997275s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.382566376s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:26:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.382596544s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.462960387s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:25:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.462997428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XX] duration: 30.00361769s trace: https://jaeger.dev.storj.io/trace/322cb0b33f6f935 FAIL (2025-12-29 18:25:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: J6IVG5NQ4JYSZ7ZB6APN2JVEF74MMLGEFOPUI34FPS74HH77ALUA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RJLQMV3FZEVPFR4EORZP7JOGC5M7YARZVY2RVRLWLQELZPCGZKGQ) context deadline exceeded ======LOGS====== Dec 29 18:25:27.495891612 (578h56m27.366974647s) [upload=507515]: Starting upload Dec 29 18:25:27.495951246 (578h56m27.367034281s) [upload=507515]: Done starting upload Dec 29 18:25:27.496025938 (578h56m27.367108973s) [upload=507515]: Uploading segments... Dec 29 18:25:27.496192846 (578h56m27.367275881s) [upload=507515]: Got next segment. Inline: false Dec 29 18:25:27.496235757 (578h56m27.367318792s) [upload=507515]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:25:27.496240937 (578h56m27.367323982s) [upload=507515]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:25:27.744036697 (578h56m27.615119742s) [seg_pos={0 0}, upload=507515]: Begin upload segment... Dec 29 18:25:27.744077726 (578h56m27.615160771s) [seg_pos={0 0}, upload=507515]: Done begin upload segment. Dec 29 18:25:27.744100990 (578h56m27.615184035s) [noise=true , address=77.90.31.15:4700 , piece_id=FSKTQLLC5UX3DHT3FVYR7NFETUYIF4YGZOM2ROD6YSKCGUD7RNGA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.744662880 (578h56m27.615745915s) [noise=true , address=145.79.227.23:4546 , piece_id=CLG6DGNNXSAEQQRGIZP22OPJINJ3POMGAT6YMDAWNKAMF5FANU5Q, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.744963633 (578h56m27.616046678s) [noise=true , address=45.91.116.13:4154 , piece_id=Z5VJ6TRVXCTHPWGDXUTXMESZQYTHA4AUAFW43JCH3N4UIUG7KTKA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.745406917 (578h56m27.616489952s) [noise=true , address=77.90.31.19:4842 , piece_id=BLSNHSS25ASTUSCQOJNZI2IE7FBRYU6COPT5NXNUPAWM3VYPWZPA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.745716988 (578h56m27.616800033s) [noise=true , address=77.90.31.10:4517 , piece_id=J6IVG5NQ4JYSZ7ZB6APN2JVEF74MMLGEFOPUI34FPS74HH77ALUA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.746137188 (578h56m27.617220223s) [noise=true , address=45.91.116.5:4300 , piece_id=S62LTREU5MH24S455F3XXS3RDIA4M6ESWNXFVJFZDHB5CJBVNIYQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.746466194 (578h56m27.617549229s) [noise=true , address=145.79.227.44:4194 , piece_id=TC3QIVFM6SGWEGCZ3Q4OFT2C5TBXMH6KOPXGCYJO6MDZHZJUPBDQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.746815499 (578h56m27.617898534s) [noise=true , address=45.91.116.12:4225 , piece_id=QA6HDQPPKCG5VB5CCKNUKHIZIVCH6EEYNHIGTSU4M4LODB5IMZCA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.747198949 (578h56m27.618281984s) [noise=true , address=45.91.116.9:4261 , piece_id=RJLQMV3FZEVPFR4EORZP7JOGC5M7YARZVY2RVRLWLQELZPCGZKGQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.747603971 (578h56m27.618687016s) [noise=true , address=145.79.227.36:4518 , piece_id=GWZESEOH2M3RHLP7YIZADMSCR2AJT34IE5UEAUPJMRNEOPL2G7WQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.747961211 (578h56m27.619044256s) [noise=true , address=45.91.116.4:4940 , piece_id=2LTKTTOISUK67GRVB34XZ2CYKJANG4YFE6G3JQMYPCMEKMLIKUEQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.748363567 (578h56m27.619446612s) [noise=true , address=145.79.227.4:4187 , piece_id=WCDZ2XYU3HWL2R67PYERAO3XL72UFGCIHLL3CNIXY5AVM3KEWGZA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.749220548 (578h56m27.620303594s) [noise=true , address=145.79.227.22:4509 , piece_id=D3VU3RLU4IDKJKAO7NYDD4EFL4MJ42TKAI3SDJIAQI5Y4ROSDMSQ, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.749499299 (578h56m27.620582344s) [noise=true , address=145.79.227.41:4085 , piece_id=LXRNUVZTQ7O3YXYFKXKWCJNHGWPYRF76FCDFTL4LHALOQ4AZBAJA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.749842964 (578h56m27.620926009s) [noise=true , address=145.79.227.17:4972 , piece_id=4C2NLVFV2V6RSL3NPJ4BAFZ2JYHWDTMACNMLJFS6TENYDH7LALDA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:27.750152183 (578h56m27.621235228s) [noise=true , address=145.79.227.51:4411 , piece_id=DHTKNBIGFPA4SCDME37BUAOLZCRLY7B35FKGDWVHJQ6WDFKB44PA, seg_pos={0 0}, upload=507515]: Uploading piece... Dec 29 18:25:29.353321468 (578h56m29.224404513s) [upload=507515]: Next returned nil segment Dec 29 18:25:29.353335826 (578h56m29.224418871s) [upload=507515]: Waiting for error group managing segments... Dec 29 18:25:29.725986288 (578h56m29.597069323s) [noise=true , address=145.79.227.4:4187 , piece_id=WCDZ2XYU3HWL2R67PYERAO3XL72UFGCIHLL3CNIXY5AVM3KEWGZA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.735253815 (578h56m29.60633685s) [noise=true , address=145.79.227.17:4972 , piece_id=4C2NLVFV2V6RSL3NPJ4BAFZ2JYHWDTMACNMLJFS6TENYDH7LALDA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.735668664 (578h56m29.606751699s) [noise=true , address=145.79.227.36:4518 , piece_id=GWZESEOH2M3RHLP7YIZADMSCR2AJT34IE5UEAUPJMRNEOPL2G7WQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.736204014 (578h56m29.60728705s) [noise=true , address=145.79.227.51:4411 , piece_id=DHTKNBIGFPA4SCDME37BUAOLZCRLY7B35FKGDWVHJQ6WDFKB44PA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.751744596 (578h56m29.622827641s) [noise=true , address=145.79.227.23:4546 , piece_id=CLG6DGNNXSAEQQRGIZP22OPJINJ3POMGAT6YMDAWNKAMF5FANU5Q, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.763374571 (578h56m29.634457616s) [noise=true , address=145.79.227.22:4509 , piece_id=D3VU3RLU4IDKJKAO7NYDD4EFL4MJ42TKAI3SDJIAQI5Y4ROSDMSQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.767005565 (578h56m29.63808861s) [noise=true , address=145.79.227.41:4085 , piece_id=LXRNUVZTQ7O3YXYFKXKWCJNHGWPYRF76FCDFTL4LHALOQ4AZBAJA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.920881859 (578h56m29.791964894s) [noise=true , address=77.90.31.10:4517 , piece_id=J6IVG5NQ4JYSZ7ZB6APN2JVEF74MMLGEFOPUI34FPS74HH77ALUA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:29.971256640 (578h56m29.842339675s) [noise=true , address=45.91.116.13:4154 , piece_id=Z5VJ6TRVXCTHPWGDXUTXMESZQYTHA4AUAFW43JCH3N4UIUG7KTKA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.096541321 (578h56m29.967624366s) [noise=true , address=45.91.116.5:4300 , piece_id=S62LTREU5MH24S455F3XXS3RDIA4M6ESWNXFVJFZDHB5CJBVNIYQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.103949577 (578h56m29.975032622s) [noise=true , address=77.90.31.15:4700 , piece_id=FSKTQLLC5UX3DHT3FVYR7NFETUYIF4YGZOM2ROD6YSKCGUD7RNGA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.104446393 (578h56m29.975529438s) [noise=true , address=45.91.116.9:4261 , piece_id=RJLQMV3FZEVPFR4EORZP7JOGC5M7YARZVY2RVRLWLQELZPCGZKGQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.751295472 (578h56m30.622378517s) [noise=true , address=45.91.116.12:4225 , piece_id=QA6HDQPPKCG5VB5CCKNUKHIZIVCH6EEYNHIGTSU4M4LODB5IMZCA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: <nil> Dec 29 18:25:30.751320850 (578h56m30.622403895s) [seg_pos={0 0}, upload=507515]: Segment reached optimal threshold of 13 pieces. Dec 29 18:25:30.751538815 (578h56m30.62262186s) [noise=true , address=77.90.31.19:4842 , piece_id=BLSNHSS25ASTUSCQOJNZI2IE7FBRYU6COPT5NXNUPAWM3VYPWZPA, seg_pos={0 0}, upload=507515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 18:25:30.751549245 (578h56m30.62263229s) [noise=true , address=77.90.31.19:4842 , piece_id=BLSNHSS25ASTUSCQOJNZI2IE7FBRYU6COPT5NXNUPAWM3VYPWZPA, seg_pos={0 0}, upload=507515]: Long tail cancellation due to optimal threshold Dec 29 18:25:30.751736482 (578h56m30.622819517s) [noise=true , address=145.79.227.44:4194 , piece_id=TC3QIVFM6SGWEGCZ3Q4OFT2C5TBXMH6KOPXGCYJO6MDZHZJUPBDQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:25:30.751758403 (578h56m30.622841458s) [noise=true , address=145.79.227.44:4194 , piece_id=TC3QIVFM6SGWEGCZ3Q4OFT2C5TBXMH6KOPXGCYJO6MDZHZJUPBDQ, seg_pos={0 0}, upload=507515]: Long tail cancellation due to optimal threshold Dec 29 18:25:30.751765436 (578h56m30.622848471s) [noise=true , address=45.91.116.4:4940 , piece_id=2LTKTTOISUK67GRVB34XZ2CYKJANG4YFE6G3JQMYPCMEKMLIKUEQ, seg_pos={0 0}, upload=507515]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 29 18:25:30.751778552 (578h56m30.622861597s) [noise=true , address=45.91.116.4:4940 , piece_id=2LTKTTOISUK67GRVB34XZ2CYKJANG4YFE6G3JQMYPCMEKMLIKUEQ, seg_pos={0 0}, upload=507515]: Long tail cancellation due to optimal threshold Dec 29 18:25:30.751799841 (578h56m30.622882877s) [seg_pos={0 0}, upload=507515]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:25:30.751885245 (578h56m30.62296829s) [upload=507515]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:25:30.751892158 (578h56m30.622975193s) [upload=507515]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:25:31.050056256 (578h56m30.921139291s) [upload=507515]: Done uploading segments... Dec 29 18:25:31.050082466 (578h56m30.921165501s) [upload=507515]: Upload finished. err: <nil> Duration: 30.003643449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.334530008s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:25:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.334552379s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.387528827s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:25:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.387555086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 30.579649143s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:25:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 30.579678671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......X] duration: 30.003197506s trace: https://jaeger.dev.storj.io/trace/5bcb545882f80b44 FAIL (2025-12-29 18:24:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: QL2VPDIXIYTBKJ3LHTMXUQG47ANBBINESBZSUYJRCFS4N5HTBLHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GWQK2PWNQQ3EOPGXNY3HITL526EPKZI2E6RKXV4F3OFQNT4VX6OQ) context deadline exceeded ======LOGS====== gments... Dec 29 18:24:13.483356826 (578h55m13.354439871s) [upload=507499]: Got next segment. Inline: false Dec 29 18:24:13.483401041 (578h55m13.354484076s) [upload=507499]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:24:13.483406441 (578h55m13.354489486s) [upload=507499]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:24:13.748310639 (578h55m13.619393684s) [seg_pos={0 0}, upload=507499]: Begin upload segment... Dec 29 18:24:13.748364862 (578h55m13.619447897s) [seg_pos={0 0}, upload=507499]: Done begin upload segment. Dec 29 18:24:13.748406632 (578h55m13.619489707s) [noise=true , address=45.91.116.8:4045 , piece_id=OMCXDTMQNC7CVXRXBHCS3NXHAEG5WHHDSKSF5FPJOGM2AWUZWHCA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.748507063 (578h55m13.619590108s) [noise=true , address=145.79.227.32:4836 , piece_id=DVD6Y54RSRSQT656SWMK32HA5MTJKYQANXVQLLQTD7UNEDECCRHQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.749078200 (578h55m13.620161245s) [noise=true , address=145.79.227.16:4909 , piece_id=XKGMGNTEIBSWHNHAUKAHR7D3DDOTO7QTBHE5KDHWH6UKNWIC6PBA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.749417587 (578h55m13.620500632s) [noise=true , address=45.91.116.7:4012 , piece_id=ZKE374IL4M5LW2KC7JNLZK7DFPBQPOYQMIPXALC2DJ3R7RUWBOFQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.749878193 (578h55m13.620961239s) [noise=true , address=45.91.116.9:4264 , piece_id=QL2VPDIXIYTBKJ3LHTMXUQG47ANBBINESBZSUYJRCFS4N5HTBLHQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.750233109 (578h55m13.621316144s) [noise=true , address=145.79.227.11:4729 , piece_id=OJ2OYREMNIIWOOB63ZSSKUXI5L6GPJQBE7LFSGE766PFOYJQWVQA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.750596111 (578h55m13.621679146s) [noise=true , address=145.79.227.50:4375 , piece_id=NA35IP4WGF5VPUPQFOKKPCK3BH5EKVEDP5WCHIHOC6OYISVNGAWQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.750958841 (578h55m13.622041886s) [noise=true , address=145.79.227.2:4113 , piece_id=KLRRZ6N6TIFNRMUO7QTAO6IEI7WOA2XWICPUQWXGV4QJKRHTR67Q, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.751381486 (578h55m13.622464521s) [noise=true , address=45.91.116.14:4190 , piece_id=GWQK2PWNQQ3EOPGXNY3HITL526EPKZI2E6RKXV4F3OFQNT4VX6OQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.752396238 (578h55m13.623479313s) [noise=true , address=45.91.116.6:4974 , piece_id=F7GY2RNNW25WV7NK2M65VOJXUAGXFYZ62QB2OXZNEN3ANLKVY5MQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.753649785 (578h55m13.62473282s) [noise=true , address=145.79.227.3:4150 , piece_id=J7TG5GXUFRQC3MNFZEWUYDLF3N3DWIQR4CYYAEYBECADW6CWNOGA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.754844190 (578h55m13.625927235s) [noise=true , address=145.79.227.6:4259 , piece_id=A73B6FI3W4257FJZ7PDYKCVOAK7FWXYOY5LAJO4MFF6WBBQEMHKQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.756837696 (578h55m13.627920731s) [noise=true , address=77.90.31.11:4556 , piece_id=6VH5X3XLYBXCA4VPWYY33Z34XJ3WEDLHVLPX4B7XYFALFN6HPKOQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.758451268 (578h55m13.629534303s) [noise=true , address=145.79.227.47:4266 , piece_id=QUMQLLHPAOBXGVT3EHGUO2PBHA3MVWSKQ6S3IISHUNECC7NU3OMA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.759642376 (578h55m13.630725411s) [noise=true , address=145.79.227.36:4518 , piece_id=LJJIQM7QENYCHZXM7H2HE3TUNCXUWM453EZNH5HRMHXLGTDGQ4CA, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:13.760881596 (578h55m13.631964641s) [noise=true , address=77.90.31.14:4661 , piece_id=5HFUCZWSDRGXXKY2XF3J4TOK4P6VKPTZ56EIIR6CTADBGVUNDXEQ, seg_pos={0 0}, upload=507499]: Uploading piece... Dec 29 18:24:15.448480408 (578h55m15.319563443s) [upload=507499]: Next returned nil segment Dec 29 18:24:15.448494124 (578h55m15.319577169s) [upload=507499]: Waiting for error group managing segments... Dec 29 18:24:15.732661511 (578h55m15.603744556s) [noise=true , address=145.79.227.32:4836 , piece_id=DVD6Y54RSRSQT656SWMK32HA5MTJKYQANXVQLLQTD7UNEDECCRHQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.741589742 (578h55m15.612672777s) [noise=true , address=145.79.227.3:4150 , piece_id=J7TG5GXUFRQC3MNFZEWUYDLF3N3DWIQR4CYYAEYBECADW6CWNOGA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.748295451 (578h55m15.619378496s) [noise=true , address=145.79.227.6:4259 , piece_id=A73B6FI3W4257FJZ7PDYKCVOAK7FWXYOY5LAJO4MFF6WBBQEMHKQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.752349250 (578h55m15.623432285s) [noise=true , address=145.79.227.47:4266 , piece_id=QUMQLLHPAOBXGVT3EHGUO2PBHA3MVWSKQ6S3IISHUNECC7NU3OMA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.756693101 (578h55m15.627776136s) [noise=true , address=145.79.227.11:4729 , piece_id=OJ2OYREMNIIWOOB63ZSSKUXI5L6GPJQBE7LFSGE766PFOYJQWVQA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.759166110 (578h55m15.630249145s) [noise=true , address=145.79.227.36:4518 , piece_id=LJJIQM7QENYCHZXM7H2HE3TUNCXUWM453EZNH5HRMHXLGTDGQ4CA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.765079070 (578h55m15.636162115s) [noise=true , address=145.79.227.50:4375 , piece_id=NA35IP4WGF5VPUPQFOKKPCK3BH5EKVEDP5WCHIHOC6OYISVNGAWQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.774799949 (578h55m15.645882984s) [noise=true , address=145.79.227.2:4113 , piece_id=KLRRZ6N6TIFNRMUO7QTAO6IEI7WOA2XWICPUQWXGV4QJKRHTR67Q, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:15.781963059 (578h55m15.653046094s) [noise=true , address=145.79.227.16:4909 , piece_id=XKGMGNTEIBSWHNHAUKAHR7D3DDOTO7QTBHE5KDHWH6UKNWIC6PBA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.034054689 (578h55m15.905137734s) [noise=true , address=45.91.116.9:4264 , piece_id=QL2VPDIXIYTBKJ3LHTMXUQG47ANBBINESBZSUYJRCFS4N5HTBLHQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.088923118 (578h55m15.960006153s) [noise=true , address=45.91.116.8:4045 , piece_id=OMCXDTMQNC7CVXRXBHCS3NXHAEG5WHHDSKSF5FPJOGM2AWUZWHCA, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.222580781 (578h55m16.093663827s) [noise=true , address=45.91.116.14:4190 , piece_id=GWQK2PWNQQ3EOPGXNY3HITL526EPKZI2E6RKXV4F3OFQNT4VX6OQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.323827731 (578h55m16.194910766s) [noise=true , address=45.91.116.6:4974 , piece_id=F7GY2RNNW25WV7NK2M65VOJXUAGXFYZ62QB2OXZNEN3ANLKVY5MQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: <nil> Dec 29 18:24:16.323854903 (578h55m16.194937948s) [seg_pos={0 0}, upload=507499]: Segment reached optimal threshold of 13 pieces. Dec 29 18:24:16.324114357 (578h55m16.195197402s) [noise=true , address=45.91.116.7:4012 , piece_id=ZKE374IL4M5LW2KC7JNLZK7DFPBQPOYQMIPXALC2DJ3R7RUWBOFQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:50706->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:50706->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50706->45.91.116.7:4012: use of closed network connection Dec 29 18:24:16.324122292 (578h55m16.195205327s) [noise=true , address=45.91.116.7:4012 , piece_id=ZKE374IL4M5LW2KC7JNLZK7DFPBQPOYQMIPXALC2DJ3R7RUWBOFQ, seg_pos={0 0}, upload=507499]: Long tail cancellation due to optimal threshold Dec 29 18:24:16.324185493 (578h55m16.195268528s) [noise=true , address=77.90.31.11:4556 , piece_id=6VH5X3XLYBXCA4VPWYY33Z34XJ3WEDLHVLPX4B7XYFALFN6HPKOQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 18:24:16.324202294 (578h55m16.195285329s) [noise=true , address=77.90.31.11:4556 , piece_id=6VH5X3XLYBXCA4VPWYY33Z34XJ3WEDLHVLPX4B7XYFALFN6HPKOQ, seg_pos={0 0}, upload=507499]: Long tail cancellation due to optimal threshold Dec 29 18:24:16.324406734 (578h55m16.195489779s) [noise=true , address=77.90.31.14:4661 , piece_id=5HFUCZWSDRGXXKY2XF3J4TOK4P6VKPTZ56EIIR6CTADBGVUNDXEQ, seg_pos={0 0}, upload=507499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:59702->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:59702->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59702->77.90.31.14:4661: use of closed network connection Dec 29 18:24:16.324413406 (578h55m16.195496441s) [noise=true , address=77.90.31.14:4661 , piece_id=5HFUCZWSDRGXXKY2XF3J4TOK4P6VKPTZ56EIIR6CTADBGVUNDXEQ, seg_pos={0 0}, upload=507499]: Long tail cancellation due to optimal threshold Dec 29 18:24:16.324432272 (578h55m16.195515307s) [seg_pos={0 0}, upload=507499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:24:16.324605092 (578h55m16.195688127s) [upload=507499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:24:16.324610652 (578h55m16.195693697s) [upload=507499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:24:16.621668113 (578h55m16.492751148s) [upload=507499]: Done uploading segments... Dec 29 18:24:16.621691057 (578h55m16.492774092s) [upload=507499]: Upload finished. err: <nil> Duration: 30.003224206s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.420088409s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:24:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420114379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 34.822215446s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:24:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 34.82224878s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.508063673s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.508161598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 27.61621003s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:23:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 27.616238933s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.541864925s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:23:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541896274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X......] duration: 6.359328271s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:23:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 6.359347457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 36.676825899s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:22:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 36.676840997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.587067693s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:22:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587109974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 37.107171863s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:21:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 37.107188535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X.....] duration: 5.727489623s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:21:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 5.727524489s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.456239945s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:21:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.456280611s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X....] duration: 8.741857335s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:21:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 8.741884737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 43.431571089s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:21:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 43.431603311s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.492615677s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:20:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492711418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X...] duration: 16.365733401s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 16.365763287s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 34.687822736s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:19:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 34.687840189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.331934144s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:19:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.331960813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 35.991350282s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:19:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.991383404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X..] duration: 6.036358077s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:19:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 6.036377814s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.37527874s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:18:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.375333775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 29.51174655s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:18:36 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 29.5117627s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X.] duration: 6.179231278s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:18:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 6.179260833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 29.219329025s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:18:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 29.219357439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.352724491s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:18:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.352754509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.X] duration: 30.003850774s trace: https://jaeger.dev.storj.io/trace/37deca3a5695bc8f FAIL (2025-12-29 18:17:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA) context deadline exceeded ======LOGS====== Dec 29 18:17:27.683505271 (578h48m27.554588307s) [upload=507405]: Starting upload Dec 29 18:17:27.683617325 (578h48m27.55470036s) [upload=507405]: Done starting upload Dec 29 18:17:27.683658934 (578h48m27.554741979s) [upload=507405]: Uploading segments... Dec 29 18:17:27.683903289 (578h48m27.554986334s) [upload=507405]: Got next segment. Inline: false Dec 29 18:17:27.683961059 (578h48m27.555044154s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:17:27.683990656 (578h48m27.555073721s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:17:27.928215687 (578h48m27.799298732s) [seg_pos={0 0}, upload=507405]: Begin upload segment... Dec 29 18:17:27.928360352 (578h48m27.799443397s) [seg_pos={0 0}, upload=507405]: Done begin upload segment. Dec 29 18:17:27.928406930 (578h48m27.799489965s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.928458439 (578h48m27.799541484s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.929193629 (578h48m27.800276674s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930052925 (578h48m27.80113597s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.930621528 (578h48m27.801704583s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.932507930 (578h48m27.803590965s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.933327320 (578h48m27.804410365s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.935719154 (578h48m27.806802199s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.938347089 (578h48m27.809430124s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.940299085 (578h48m27.81138212s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.941852723 (578h48m27.812935768s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.943326871 (578h48m27.814409916s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.944782522 (578h48m27.815865567s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946438475 (578h48m27.817521521s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.946869305 (578h48m27.817952341s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:27.949711929 (578h48m27.820794964s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Uploading piece... Dec 29 18:17:29.632497317 (578h48m29.503580352s) [upload=507405]: Next returned nil segment Dec 29 18:17:29.632515101 (578h48m29.503598176s) [upload=507405]: Waiting for error group managing segments... Dec 29 18:17:29.916807286 (578h48m29.787890331s) [noise=true , address=145.79.227.15:4876 , piece_id=VNMG5YLULPEZNVIQHQSKHFGYKI5BOK73VDJ3RH3IP2AL6OCSL6YA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.926380194 (578h48m29.797463239s) [noise=true , address=145.79.227.48:4301 , piece_id=7GEHPRYKEVKETCGK2ACP4SXREPGADAISGBFKCMJZA4KPMRD5LFVQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.927162914 (578h48m29.798245959s) [noise=true , address=145.79.227.7:4294 , piece_id=FJTE6YKBNTW6EYVCMZQVIZGFHTEQEHSYRGV5IAD3KT2HPBSP7TOA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.932570482 (578h48m29.803653517s) [noise=true , address=145.79.227.16:4909 , piece_id=F4BDLPLAB5CHFI3ZLNOESYRYT2JOZAQN4NHZZWPW2MMDWEB6WIMQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.934594115 (578h48m29.80567715s) [noise=true , address=145.79.227.45:4229 , piece_id=BESFHMH36I6IYAL6WOA2EKTYQZX34Y7DT4AFLWLCDU4XK5N5FKAA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.939015764 (578h48m29.810098809s) [noise=true , address=145.79.227.3:4150 , piece_id=YLE7WL5MPSHL3CLPK6TEB2J5V6KOMCZOE3YXZN4SLSMSSKPYNXLA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.942541197 (578h48m29.813624232s) [noise=true , address=145.79.227.41:4085 , piece_id=OLV4NFXTTYUDUREOKFIZ2ZWHT3COA4VUUMLRNX5GCBBFECICDE6A, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.944926970 (578h48m29.816010015s) [noise=true , address=145.79.227.38:4978 , piece_id=3LMCOS6A37R2VE5RDFJPGXATSWRNGPKNCUVALWZCS5DQILKHLBLQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.948098018 (578h48m29.819181053s) [noise=true , address=145.79.227.47:4266 , piece_id=VXRPUKVOR4GSONRRU32Z5B6LERKHZEXSQYN7GE4GUSL5I7MDFBIA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:29.949916971 (578h48m29.821000016s) [noise=true , address=145.79.227.21:4475 , piece_id=XKXPWHJ5EK6PV5ZIAURBQLBYCTILTDOMDUQRSCKGXJJMYWY2SIGA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.194442785 (578h48m30.06552583s) [noise=true , address=45.91.116.14:4191 , piece_id=I37X76UBPDCOLVKJ3S3DN6DDLAG2SRZUFTNPRBVWZGBWU7R7WAMA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.289892726 (578h48m30.160975781s) [noise=true , address=45.91.116.5:4298 , piece_id=OBPHNO5AAGGYDYJJE225REYYSE6CE3JXBP77N5TLXYS5OPJF6SJA, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528875687 (578h48m30.399958722s) [noise=true , address=45.91.116.11:4119 , piece_id=P56KA7GC5FPT6SR7Q3BJFULNY2RG2OLX3CKNWHLVZT4YYCJCIMHQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: <nil> Dec 29 18:17:30.528894383 (578h48m30.399977418s) [seg_pos={0 0}, upload=507405]: Segment reached optimal threshold of 13 pieces. Dec 29 18:17:30.529105565 (578h48m30.4001886s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529111987 (578h48m30.400195032s) [noise=true , address=45.91.116.10:4084 , piece_id=UFPYR4FU3LO5V5NZL2CIDU3OHSOMEVRJYVPXB2QOMXAUZ2YU3UYQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529327357 (578h48m30.400410392s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529345903 (578h48m30.400428948s) [noise=true , address=145.79.227.30:4762 , piece_id=HFJHLFMIIZ2XQRW7KHW337MS5KF4E4VLHE3U667OGZGCPTDTK36Q, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529360069 (578h48m30.400443104s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:17:30.529460661 (578h48m30.400543726s) [noise=true , address=145.79.227.29:4942 , piece_id=YTCXRRZ3B3W2NZLPCASJ5I7F7ASQWWKRWIH6HPD3OT5ABX4R5XSQ, seg_pos={0 0}, upload=507405]: Long tail cancellation due to optimal threshold Dec 29 18:17:30.529874990 (578h48m30.400958025s) [seg_pos={0 0}, upload=507405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:17:30.529955914 (578h48m30.401038949s) [upload=507405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:17:30.529961635 (578h48m30.40104468s) [upload=507405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:17:30.822776598 (578h48m30.693859643s) [upload=507405]: Done uploading segments... Dec 29 18:17:30.822800073 (578h48m30.693883118s) [upload=507405]: Upload finished. err: <nil> Duration: 30.003870041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 15.36985612s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:17:33 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.369887631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.541623633s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:17:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541681813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.598978206s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.599002432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 36.630812997s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:16:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 36.630841131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX.] duration: 5.969012289s trace: https://jaeger.dev.storj.io/trace/15fe71347bf9ee00 FAIL (2025-12-29 18:16:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: J4SYL2F7XEOIPZ7346KAIH6CEIIL4KVA6GH36GJU3ORH3IFX53NQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 5NNXPYPYX3GMDLOBQT4YWUBQHIYN54QYDFK73XJRE6YXXEZWZKLQ) context deadline exceeded ======LOGS====== 18:15:52.543384310 (578h46m52.414467345s) [upload=507382]: Starting upload Dec 29 18:15:52.543497717 (578h46m52.414580752s) [upload=507382]: Done starting upload Dec 29 18:15:52.543597926 (578h46m52.414680962s) [upload=507382]: Uploading segments... Dec 29 18:15:52.543752431 (578h46m52.414835476s) [upload=507382]: Got next segment. Inline: false Dec 29 18:15:52.543812556 (578h46m52.414895591s) [upload=507382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:15:52.543819840 (578h46m52.414902875s) [upload=507382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:15:52.832834727 (578h46m52.703917783s) [seg_pos={0 0}, upload=507382]: Begin upload segment... Dec 29 18:15:52.832887928 (578h46m52.703970973s) [seg_pos={0 0}, upload=507382]: Done begin upload segment. Dec 29 18:15:52.832922614 (578h46m52.704005659s) [noise=true , address=77.90.31.12:4590 , piece_id=FFH4YLYQCXI4YTEHOC3YOEJ5GP6RAVE27UHAJXLJ3VDAVWBK5DQQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.832944366 (578h46m52.704027411s) [noise=true , address=77.90.31.13:4626 , piece_id=J4SYL2F7XEOIPZ7346KAIH6CEIIL4KVA6GH36GJU3ORH3IFX53NQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.833478653 (578h46m52.704561688s) [noise=true , address=145.79.227.23:4546 , piece_id=L6VBLLE2EUHMWY6HY3YHYHL4K3KYSSYFWTNKY3OSI4C4FMQCS3AQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.833854439 (578h46m52.704937474s) [noise=true , address=145.79.227.37:4565 , piece_id=KQBIWDPNXANDWRZH6NAGIFSGMX3SRP6B3ZVNKRIOYFWYOJQIY5UQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.834135464 (578h46m52.705218499s) [noise=true , address=145.79.227.49:4339 , piece_id=GNT7JANPGM2WV5F2APJ6CT5WRHRZ4VVXTKD45GA6BAFKIO3RSYIA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.834445695 (578h46m52.70552875s) [noise=true , address=145.79.227.21:4475 , piece_id=2WVOLALKXPPY3JTAHXZMY4JH2VRANN7EH7WFIBU3UU4EHGPZSH5A, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.834774331 (578h46m52.705857366s) [noise=true , address=45.91.116.9:4264 , piece_id=QESABYWELUXMYDAUO4WRYN3XUL43RVTARYRB7C36T7ZOJJDS76GQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.835061087 (578h46m52.706144122s) [noise=true , address=145.79.227.14:4838 , piece_id=ITKZHDDHT2SDXUT53MWJSNKH75IJFZE2ENSCOK3XKMLOYMMRNFFQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.835384162 (578h46m52.706467197s) [noise=true , address=145.79.227.41:4086 , piece_id=HTHFW2CFICWLBUYHKI6ENWQUXFH5PXKZPK4QUQ4FYGWA2LHBMZYA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.835658755 (578h46m52.70674179s) [noise=true , address=45.91.116.8:4045 , piece_id=5NNXPYPYX3GMDLOBQT4YWUBQHIYN54QYDFK73XJRE6YXXEZWZKLQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.838006745 (578h46m52.70908978s) [noise=true , address=145.79.227.46:4601 , piece_id=2H6QRNN6HHQWO5JF6JACVYVY45X32N4J5QAPV3UVNTKH6QVGEYCA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.838704714 (578h46m52.709787749s) [noise=true , address=145.79.227.39:4013 , piece_id=GXEAHU26PFXINKCPU53UK4QWQZLZVUEO3TMHTAKK7X5ABZDZ25RA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.839233502 (578h46m52.710316547s) [noise=true , address=145.79.227.50:4374 , piece_id=A4NT67NE5C453LDZ3EAR6PUCBG7CGYWFMXQ2JMAJZMRV64Q3PDZQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.840726385 (578h46m52.71180942s) [noise=true , address=145.79.227.22:4510 , piece_id=DVEHVREXJZNYY6WYNTP3CSRXPANFXAB56AQ3D7GK6SVCEYHMX66Q, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.842302716 (578h46m52.713385751s) [noise=true , address=145.79.227.16:4912 , piece_id=F4JGMQCWVRQJY5DYHXC5COMVWMODS572IIAACJ6E47NFQSU44AVA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.843900509 (578h46m52.714983544s) [noise=true , address=145.79.227.48:4303 , piece_id=QKRSBEZ74GNRD5XQYQSOHZHAJYLJFJMLQNRWU7QPRHUJQE4KADWA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:54.436174848 (578h46m54.307257893s) [upload=507382]: Next returned nil segment Dec 29 18:15:54.436196660 (578h46m54.307279695s) [upload=507382]: Waiting for error group managing segments... Dec 29 18:15:54.819727270 (578h46m54.690810335s) [noise=true , address=145.79.227.37:4565 , piece_id=KQBIWDPNXANDWRZH6NAGIFSGMX3SRP6B3ZVNKRIOYFWYOJQIY5UQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.828863638 (578h46m54.699946683s) [noise=true , address=145.79.227.14:4838 , piece_id=ITKZHDDHT2SDXUT53MWJSNKH75IJFZE2ENSCOK3XKMLOYMMRNFFQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.829522912 (578h46m54.700605957s) [noise=true , address=145.79.227.22:4510 , piece_id=DVEHVREXJZNYY6WYNTP3CSRXPANFXAB56AQ3D7GK6SVCEYHMX66Q, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.830648987 (578h46m54.701732032s) [noise=true , address=145.79.227.39:4013 , piece_id=GXEAHU26PFXINKCPU53UK4QWQZLZVUEO3TMHTAKK7X5ABZDZ25RA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.832007504 (578h46m54.703090549s) [noise=true , address=145.79.227.41:4086 , piece_id=HTHFW2CFICWLBUYHKI6ENWQUXFH5PXKZPK4QUQ4FYGWA2LHBMZYA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.833398713 (578h46m54.704481748s) [noise=true , address=145.79.227.21:4475 , piece_id=2WVOLALKXPPY3JTAHXZMY4JH2VRANN7EH7WFIBU3UU4EHGPZSH5A, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.835211004 (578h46m54.706294049s) [noise=true , address=145.79.227.16:4912 , piece_id=F4JGMQCWVRQJY5DYHXC5COMVWMODS572IIAACJ6E47NFQSU44AVA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.835650300 (578h46m54.706733335s) [noise=true , address=145.79.227.46:4601 , piece_id=2H6QRNN6HHQWO5JF6JACVYVY45X32N4J5QAPV3UVNTKH6QVGEYCA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.841786233 (578h46m54.712869278s) [noise=true , address=145.79.227.50:4374 , piece_id=A4NT67NE5C453LDZ3EAR6PUCBG7CGYWFMXQ2JMAJZMRV64Q3PDZQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.848493405 (578h46m54.71957644s) [noise=true , address=145.79.227.48:4303 , piece_id=QKRSBEZ74GNRD5XQYQSOHZHAJYLJFJMLQNRWU7QPRHUJQE4KADWA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.007266402 (578h46m54.878349437s) [noise=true , address=77.90.31.13:4626 , piece_id=J4SYL2F7XEOIPZ7346KAIH6CEIIL4KVA6GH36GJU3ORH3IFX53NQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.166111307 (578h46m55.037194353s) [noise=true , address=45.91.116.8:4045 , piece_id=5NNXPYPYX3GMDLOBQT4YWUBQHIYN54QYDFK73XJRE6YXXEZWZKLQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.217868551 (578h46m55.088951586s) [noise=true , address=45.91.116.9:4264 , piece_id=QESABYWELUXMYDAUO4WRYN3XUL43RVTARYRB7C36T7ZOJJDS76GQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.217892256 (578h46m55.088975301s) [seg_pos={0 0}, upload=507382]: Segment reached optimal threshold of 13 pieces. Dec 29 18:15:55.218099601 (578h46m55.089182646s) [noise=true , address=77.90.31.12:4590 , piece_id=FFH4YLYQCXI4YTEHOC3YOEJ5GP6RAVE27UHAJXLJ3VDAVWBK5DQQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 18:15:55.218106144 (578h46m55.089189189s) [noise=true , address=77.90.31.12:4590 , piece_id=FFH4YLYQCXI4YTEHOC3YOEJ5GP6RAVE27UHAJXLJ3VDAVWBK5DQQ, seg_pos={0 0}, upload=507382]: Long tail cancellation due to optimal threshold Dec 29 18:15:55.218255017 (578h46m55.089338062s) [noise=true , address=145.79.227.23:4546 , piece_id=L6VBLLE2EUHMWY6HY3YHYHL4K3KYSSYFWTNKY3OSI4C4FMQCS3AQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:36092->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:36092->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36092->145.79.227.23:4546: use of closed network connection Dec 29 18:15:55.218262692 (578h46m55.089345727s) [noise=true , address=145.79.227.23:4546 , piece_id=L6VBLLE2EUHMWY6HY3YHYHL4K3KYSSYFWTNKY3OSI4C4FMQCS3AQ, seg_pos={0 0}, upload=507382]: Long tail cancellation due to optimal threshold Dec 29 18:15:55.218272740 (578h46m55.089355785s) [noise=true , address=145.79.227.49:4339 , piece_id=GNT7JANPGM2WV5F2APJ6CT5WRHRZ4VVXTKD45GA6BAFKIO3RSYIA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 18:15:55.218288942 (578h46m55.089371987s) [noise=true , address=145.79.227.49:4339 , piece_id=GNT7JANPGM2WV5F2APJ6CT5WRHRZ4VVXTKD45GA6BAFKIO3RSYIA, seg_pos={0 0}, upload=507382]: Long tail cancellation due to optimal threshold Dec 29 18:15:55.218316945 (578h46m55.08939999s) [seg_pos={0 0}, upload=507382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:15:55.218477400 (578h46m55.089560435s) [upload=507382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:15:55.218484594 (578h46m55.089567639s) [upload=507382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:15:55.522943319 (578h46m55.394026354s) [upload=507382]: Done uploading segments... Dec 29 18:15:55.522963938 (578h46m55.394046973s) [upload=507382]: Upload finished. err: <nil> Duration: 5.969029863s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.470645785s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:16:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470671604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...XX] duration: 30.001984866s trace: https://jaeger.dev.storj.io/trace/15fe71347bf9ee00 FAIL (2025-12-29 18:16:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: J4SYL2F7XEOIPZ7346KAIH6CEIIL4KVA6GH36GJU3ORH3IFX53NQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 5NNXPYPYX3GMDLOBQT4YWUBQHIYN54QYDFK73XJRE6YXXEZWZKLQ) context deadline exceeded ======LOGS====== 18:15:52.543384310 (578h46m52.414467345s) [upload=507382]: Starting upload Dec 29 18:15:52.543497717 (578h46m52.414580752s) [upload=507382]: Done starting upload Dec 29 18:15:52.543597926 (578h46m52.414680962s) [upload=507382]: Uploading segments... Dec 29 18:15:52.543752431 (578h46m52.414835476s) [upload=507382]: Got next segment. Inline: false Dec 29 18:15:52.543812556 (578h46m52.414895591s) [upload=507382]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:15:52.543819840 (578h46m52.414902875s) [upload=507382]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:15:52.832834727 (578h46m52.703917783s) [seg_pos={0 0}, upload=507382]: Begin upload segment... Dec 29 18:15:52.832887928 (578h46m52.703970973s) [seg_pos={0 0}, upload=507382]: Done begin upload segment. Dec 29 18:15:52.832922614 (578h46m52.704005659s) [noise=true , address=77.90.31.12:4590 , piece_id=FFH4YLYQCXI4YTEHOC3YOEJ5GP6RAVE27UHAJXLJ3VDAVWBK5DQQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.832944366 (578h46m52.704027411s) [noise=true , address=77.90.31.13:4626 , piece_id=J4SYL2F7XEOIPZ7346KAIH6CEIIL4KVA6GH36GJU3ORH3IFX53NQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.833478653 (578h46m52.704561688s) [noise=true , address=145.79.227.23:4546 , piece_id=L6VBLLE2EUHMWY6HY3YHYHL4K3KYSSYFWTNKY3OSI4C4FMQCS3AQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.833854439 (578h46m52.704937474s) [noise=true , address=145.79.227.37:4565 , piece_id=KQBIWDPNXANDWRZH6NAGIFSGMX3SRP6B3ZVNKRIOYFWYOJQIY5UQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.834135464 (578h46m52.705218499s) [noise=true , address=145.79.227.49:4339 , piece_id=GNT7JANPGM2WV5F2APJ6CT5WRHRZ4VVXTKD45GA6BAFKIO3RSYIA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.834445695 (578h46m52.70552875s) [noise=true , address=145.79.227.21:4475 , piece_id=2WVOLALKXPPY3JTAHXZMY4JH2VRANN7EH7WFIBU3UU4EHGPZSH5A, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.834774331 (578h46m52.705857366s) [noise=true , address=45.91.116.9:4264 , piece_id=QESABYWELUXMYDAUO4WRYN3XUL43RVTARYRB7C36T7ZOJJDS76GQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.835061087 (578h46m52.706144122s) [noise=true , address=145.79.227.14:4838 , piece_id=ITKZHDDHT2SDXUT53MWJSNKH75IJFZE2ENSCOK3XKMLOYMMRNFFQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.835384162 (578h46m52.706467197s) [noise=true , address=145.79.227.41:4086 , piece_id=HTHFW2CFICWLBUYHKI6ENWQUXFH5PXKZPK4QUQ4FYGWA2LHBMZYA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.835658755 (578h46m52.70674179s) [noise=true , address=45.91.116.8:4045 , piece_id=5NNXPYPYX3GMDLOBQT4YWUBQHIYN54QYDFK73XJRE6YXXEZWZKLQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.838006745 (578h46m52.70908978s) [noise=true , address=145.79.227.46:4601 , piece_id=2H6QRNN6HHQWO5JF6JACVYVY45X32N4J5QAPV3UVNTKH6QVGEYCA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.838704714 (578h46m52.709787749s) [noise=true , address=145.79.227.39:4013 , piece_id=GXEAHU26PFXINKCPU53UK4QWQZLZVUEO3TMHTAKK7X5ABZDZ25RA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.839233502 (578h46m52.710316547s) [noise=true , address=145.79.227.50:4374 , piece_id=A4NT67NE5C453LDZ3EAR6PUCBG7CGYWFMXQ2JMAJZMRV64Q3PDZQ, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.840726385 (578h46m52.71180942s) [noise=true , address=145.79.227.22:4510 , piece_id=DVEHVREXJZNYY6WYNTP3CSRXPANFXAB56AQ3D7GK6SVCEYHMX66Q, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.842302716 (578h46m52.713385751s) [noise=true , address=145.79.227.16:4912 , piece_id=F4JGMQCWVRQJY5DYHXC5COMVWMODS572IIAACJ6E47NFQSU44AVA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:52.843900509 (578h46m52.714983544s) [noise=true , address=145.79.227.48:4303 , piece_id=QKRSBEZ74GNRD5XQYQSOHZHAJYLJFJMLQNRWU7QPRHUJQE4KADWA, seg_pos={0 0}, upload=507382]: Uploading piece... Dec 29 18:15:54.436174848 (578h46m54.307257893s) [upload=507382]: Next returned nil segment Dec 29 18:15:54.436196660 (578h46m54.307279695s) [upload=507382]: Waiting for error group managing segments... Dec 29 18:15:54.819727270 (578h46m54.690810335s) [noise=true , address=145.79.227.37:4565 , piece_id=KQBIWDPNXANDWRZH6NAGIFSGMX3SRP6B3ZVNKRIOYFWYOJQIY5UQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.828863638 (578h46m54.699946683s) [noise=true , address=145.79.227.14:4838 , piece_id=ITKZHDDHT2SDXUT53MWJSNKH75IJFZE2ENSCOK3XKMLOYMMRNFFQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.829522912 (578h46m54.700605957s) [noise=true , address=145.79.227.22:4510 , piece_id=DVEHVREXJZNYY6WYNTP3CSRXPANFXAB56AQ3D7GK6SVCEYHMX66Q, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.830648987 (578h46m54.701732032s) [noise=true , address=145.79.227.39:4013 , piece_id=GXEAHU26PFXINKCPU53UK4QWQZLZVUEO3TMHTAKK7X5ABZDZ25RA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.832007504 (578h46m54.703090549s) [noise=true , address=145.79.227.41:4086 , piece_id=HTHFW2CFICWLBUYHKI6ENWQUXFH5PXKZPK4QUQ4FYGWA2LHBMZYA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.833398713 (578h46m54.704481748s) [noise=true , address=145.79.227.21:4475 , piece_id=2WVOLALKXPPY3JTAHXZMY4JH2VRANN7EH7WFIBU3UU4EHGPZSH5A, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.835211004 (578h46m54.706294049s) [noise=true , address=145.79.227.16:4912 , piece_id=F4JGMQCWVRQJY5DYHXC5COMVWMODS572IIAACJ6E47NFQSU44AVA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.835650300 (578h46m54.706733335s) [noise=true , address=145.79.227.46:4601 , piece_id=2H6QRNN6HHQWO5JF6JACVYVY45X32N4J5QAPV3UVNTKH6QVGEYCA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.841786233 (578h46m54.712869278s) [noise=true , address=145.79.227.50:4374 , piece_id=A4NT67NE5C453LDZ3EAR6PUCBG7CGYWFMXQ2JMAJZMRV64Q3PDZQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:54.848493405 (578h46m54.71957644s) [noise=true , address=145.79.227.48:4303 , piece_id=QKRSBEZ74GNRD5XQYQSOHZHAJYLJFJMLQNRWU7QPRHUJQE4KADWA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.007266402 (578h46m54.878349437s) [noise=true , address=77.90.31.13:4626 , piece_id=J4SYL2F7XEOIPZ7346KAIH6CEIIL4KVA6GH36GJU3ORH3IFX53NQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.166111307 (578h46m55.037194353s) [noise=true , address=45.91.116.8:4045 , piece_id=5NNXPYPYX3GMDLOBQT4YWUBQHIYN54QYDFK73XJRE6YXXEZWZKLQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.217868551 (578h46m55.088951586s) [noise=true , address=45.91.116.9:4264 , piece_id=QESABYWELUXMYDAUO4WRYN3XUL43RVTARYRB7C36T7ZOJJDS76GQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: <nil> Dec 29 18:15:55.217892256 (578h46m55.088975301s) [seg_pos={0 0}, upload=507382]: Segment reached optimal threshold of 13 pieces. Dec 29 18:15:55.218099601 (578h46m55.089182646s) [noise=true , address=77.90.31.12:4590 , piece_id=FFH4YLYQCXI4YTEHOC3YOEJ5GP6RAVE27UHAJXLJ3VDAVWBK5DQQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 18:15:55.218106144 (578h46m55.089189189s) [noise=true , address=77.90.31.12:4590 , piece_id=FFH4YLYQCXI4YTEHOC3YOEJ5GP6RAVE27UHAJXLJ3VDAVWBK5DQQ, seg_pos={0 0}, upload=507382]: Long tail cancellation due to optimal threshold Dec 29 18:15:55.218255017 (578h46m55.089338062s) [noise=true , address=145.79.227.23:4546 , piece_id=L6VBLLE2EUHMWY6HY3YHYHL4K3KYSSYFWTNKY3OSI4C4FMQCS3AQ, seg_pos={0 0}, upload=507382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:36092->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:36092->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36092->145.79.227.23:4546: use of closed network connection Dec 29 18:15:55.218262692 (578h46m55.089345727s) [noise=true , address=145.79.227.23:4546 , piece_id=L6VBLLE2EUHMWY6HY3YHYHL4K3KYSSYFWTNKY3OSI4C4FMQCS3AQ, seg_pos={0 0}, upload=507382]: Long tail cancellation due to optimal threshold Dec 29 18:15:55.218272740 (578h46m55.089355785s) [noise=true , address=145.79.227.49:4339 , piece_id=GNT7JANPGM2WV5F2APJ6CT5WRHRZ4VVXTKD45GA6BAFKIO3RSYIA, seg_pos={0 0}, upload=507382]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 18:15:55.218288942 (578h46m55.089371987s) [noise=true , address=145.79.227.49:4339 , piece_id=GNT7JANPGM2WV5F2APJ6CT5WRHRZ4VVXTKD45GA6BAFKIO3RSYIA, seg_pos={0 0}, upload=507382]: Long tail cancellation due to optimal threshold Dec 29 18:15:55.218316945 (578h46m55.08939999s) [seg_pos={0 0}, upload=507382]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:15:55.218477400 (578h46m55.089560435s) [upload=507382]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:15:55.218484594 (578h46m55.089567639s) [upload=507382]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:15:55.522943319 (578h46m55.394026354s) [upload=507382]: Done uploading segments... Dec 29 18:15:55.522963938 (578h46m55.394046973s) [upload=507382]: Upload finished. err: <nil> Duration: 30.002015996s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.519295056s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:16:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.519326418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.X..X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...X] duration: 30.002716359s trace: https://jaeger.dev.storj.io/trace/74c22baa51edd243 FAIL (2025-12-29 18:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:15:22.279489840 (578h46m22.150572885s) [upload=507375]: Starting upload Dec 29 18:15:22.279550706 (578h46m22.150633751s) [upload=507375]: Done starting upload Dec 29 18:15:22.279610289 (578h46m22.150693324s) [upload=507375]: Uploading segments... Dec 29 18:15:22.279835348 (578h46m22.150918393s) [upload=507375]: Got next segment. Inline: false Dec 29 18:15:22.279870495 (578h46m22.15095353s) [upload=507375]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:15:22.279877247 (578h46m22.150960282s) [upload=507375]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:15:22.523225749 (578h46m22.394308784s) [seg_pos={0 0}, upload=507375]: Begin upload segment... Dec 29 18:15:22.523284371 (578h46m22.394367416s) [seg_pos={0 0}, upload=507375]: Done begin upload segment. Dec 29 18:15:22.523326752 (578h46m22.394409787s) [noise=true , address=77.90.31.14:4662 , piece_id=RHTVVYQINWHAAPUVLDRE7WH6G2O5ER4K55BGVEQWDHKG3VZKGP4Q, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.523345216 (578h46m22.394428261s) [noise=true , address=145.79.227.48:4303 , piece_id=AHUCAHYB3RZA4366CYKHBE5M6JJNR62RVLN5ANZCVZPKF2CNTIHQ, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.523854687 (578h46m22.394937732s) [noise=true , address=45.91.116.13:4155 , piece_id=2CQHJQ3U3XUPPTQ5EDNB34SJF573NNHLQNIYSXSKIJI43YM65MKA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.524472153 (578h46m22.395555188s) [noise=true , address=145.79.227.2:4113 , piece_id=K2Q5RFRYAXX2DZDAI65EKLUFN5YCZOOZKIZPRP4S7OY4DPGFG7XQ, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.524504915 (578h46m22.39558795s) [noise=true , address=77.90.31.13:4626 , piece_id=5BSSXUQXDT5EBMNB4VAAYT7VN4BTVNUJT4OLURPUTFN7RSHOAKGA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.524862215 (578h46m22.39594525s) [noise=true , address=145.79.227.50:4374 , piece_id=I25O6ZBVVN2P3BVOZLXTGXTL5PUCQVU7DV3GZMB7XQLPB7B44JTQ, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.524932239 (578h46m22.396015284s) [noise=true , address=145.79.227.19:4078 , piece_id=HCG3U7J7DPA6UFOESUYLTC7W3X5FRNAFB7YFZW365ZOQLUYCMQZQ, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.525216540 (578h46m22.396299575s) [noise=true , address=45.91.116.4:4939 , piece_id=5UBCM3YTB2C5WLEDQNH2DY25C4LIV2GE7BEOWELNT5DPJYMJSXMA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.525347049 (578h46m22.396430084s) [noise=true , address=45.91.116.8:4046 , piece_id=WFTQOJRV4GYM3EUA6C5BEBT2JTPEDH2KJDWMFA3W3XYEH5H2SXLA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.525626871 (578h46m22.396709917s) [noise=true , address=145.79.227.11:4729 , piece_id=WALUKICL7HC6SXCR3F4NW3OPT5KFYUTQ2WPZQAQOFMMASALT5GNQ, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.525979643 (578h46m22.397062688s) [noise=true , address=77.90.31.10:4519 , piece_id=4B2NII7WP2TZYNMGOGR4GDLHFXUZD4Z5M52ZJVHQKGQ6F7OC5VSA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.526018587 (578h46m22.397101632s) [noise=true , address=145.79.227.45:4231 , piece_id=S322HD7CR2YBTF7HGYEJISNQQOMZTMF6MDDCE7LQI6UOADYYAOYA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.526361089 (578h46m22.397444134s) [noise=true , address=145.79.227.16:4910 , piece_id=WRUFQHQCOR5L76TMYBPBDMKORRV7VXESG2DWVSGN37DLBZN4N3QA, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.526741153 (578h46m22.397824188s) [noise=true , address=145.79.227.52:4445 , piece_id=42PAZ4M3B7APLOLHSG6EWRRWCOJVEEU76Z2R4YTWOZ4XJ4R3ZM5Q, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.527087763 (578h46m22.398170808s) [noise=true , address=45.91.116.5:4298 , piece_id=XGCVXIJWW7LLSKMMQKIUU2O2SXRGAPW7MYEEIYUODSBW4LBKOL3Q, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:22.530061836 (578h46m22.401144941s) [noise=true , address=145.79.227.42:4121 , piece_id=DDKH47MP4LPV26WEYVSXW6SM4ALY4CAAADRBUV6XDNEMNDMYZL7Q, seg_pos={0 0}, upload=507375]: Uploading piece... Dec 29 18:15:24.223260302 (578h46m24.094343367s) [upload=507375]: Next returned nil segment Dec 29 18:15:24.223279188 (578h46m24.094362223s) [upload=507375]: Waiting for error group managing segments... Dec 29 18:15:24.494876491 (578h46m24.365959536s) [noise=true , address=145.79.227.19:4078 , piece_id=HCG3U7J7DPA6UFOESUYLTC7W3X5FRNAFB7YFZW365ZOQLUYCMQZQ, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.507522470 (578h46m24.378605505s) [noise=true , address=145.79.227.48:4303 , piece_id=AHUCAHYB3RZA4366CYKHBE5M6JJNR62RVLN5ANZCVZPKF2CNTIHQ, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.514136595 (578h46m24.38521964s) [noise=true , address=145.79.227.11:4729 , piece_id=WALUKICL7HC6SXCR3F4NW3OPT5KFYUTQ2WPZQAQOFMMASALT5GNQ, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.516674187 (578h46m24.387757232s) [noise=true , address=145.79.227.45:4231 , piece_id=S322HD7CR2YBTF7HGYEJISNQQOMZTMF6MDDCE7LQI6UOADYYAOYA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.524469099 (578h46m24.395552144s) [noise=true , address=145.79.227.16:4910 , piece_id=WRUFQHQCOR5L76TMYBPBDMKORRV7VXESG2DWVSGN37DLBZN4N3QA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.526252034 (578h46m24.397335069s) [noise=true , address=145.79.227.52:4445 , piece_id=42PAZ4M3B7APLOLHSG6EWRRWCOJVEEU76Z2R4YTWOZ4XJ4R3ZM5Q, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.541319074 (578h46m24.412402109s) [noise=true , address=145.79.227.50:4374 , piece_id=I25O6ZBVVN2P3BVOZLXTGXTL5PUCQVU7DV3GZMB7XQLPB7B44JTQ, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.542508238 (578h46m24.413591283s) [noise=true , address=145.79.227.42:4121 , piece_id=DDKH47MP4LPV26WEYVSXW6SM4ALY4CAAADRBUV6XDNEMNDMYZL7Q, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.546163919 (578h46m24.417246964s) [noise=true , address=145.79.227.2:4113 , piece_id=K2Q5RFRYAXX2DZDAI65EKLUFN5YCZOOZKIZPRP4S7OY4DPGFG7XQ, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.774528209 (578h46m24.645611254s) [noise=true , address=45.91.116.5:4298 , piece_id=XGCVXIJWW7LLSKMMQKIUU2O2SXRGAPW7MYEEIYUODSBW4LBKOL3Q, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.787906203 (578h46m24.658989248s) [noise=true , address=77.90.31.13:4626 , piece_id=5BSSXUQXDT5EBMNB4VAAYT7VN4BTVNUJT4OLURPUTFN7RSHOAKGA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.862250255 (578h46m24.73333329s) [noise=true , address=45.91.116.8:4046 , piece_id=WFTQOJRV4GYM3EUA6C5BEBT2JTPEDH2KJDWMFA3W3XYEH5H2SXLA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.885938962 (578h46m24.757022017s) [noise=true , address=45.91.116.4:4939 , piece_id=5UBCM3YTB2C5WLEDQNH2DY25C4LIV2GE7BEOWELNT5DPJYMJSXMA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: <nil> Dec 29 18:15:24.885964331 (578h46m24.757047386s) [seg_pos={0 0}, upload=507375]: Segment reached optimal threshold of 13 pieces. Dec 29 18:15:24.886211271 (578h46m24.757294316s) [noise=true , address=45.91.116.13:4155 , piece_id=2CQHJQ3U3XUPPTQ5EDNB34SJF573NNHLQNIYSXSKIJI43YM65MKA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:15:24.886222332 (578h46m24.757305377s) [noise=true , address=45.91.116.13:4155 , piece_id=2CQHJQ3U3XUPPTQ5EDNB34SJF573NNHLQNIYSXSKIJI43YM65MKA, seg_pos={0 0}, upload=507375]: Long tail cancellation due to optimal threshold Dec 29 18:15:24.886608887 (578h46m24.757691933s) [noise=true , address=77.90.31.14:4662 , piece_id=RHTVVYQINWHAAPUVLDRE7WH6G2O5ER4K55BGVEQWDHKG3VZKGP4Q, seg_pos={0 0}, upload=507375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: read tcp 188.166.225.170:60950->77.90.31.14:4662: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 18:15:24.886618497 (578h46m24.757701532s) [noise=true , address=77.90.31.14:4662 , piece_id=RHTVVYQINWHAAPUVLDRE7WH6G2O5ER4K55BGVEQWDHKG3VZKGP4Q, seg_pos={0 0}, upload=507375]: Long tail cancellation due to optimal threshold Dec 29 18:15:24.886710962 (578h46m24.757794007s) [noise=true , address=77.90.31.10:4519 , piece_id=4B2NII7WP2TZYNMGOGR4GDLHFXUZD4Z5M52ZJVHQKGQ6F7OC5VSA, seg_pos={0 0}, upload=507375]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: read tcp 188.166.225.170:50818->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 18:15:24.886730550 (578h46m24.757813595s) [noise=true , address=77.90.31.10:4519 , piece_id=4B2NII7WP2TZYNMGOGR4GDLHFXUZD4Z5M52ZJVHQKGQ6F7OC5VSA, seg_pos={0 0}, upload=507375]: Long tail cancellation due to optimal threshold Dec 29 18:15:24.886794502 (578h46m24.757877537s) [seg_pos={0 0}, upload=507375]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:15:24.886918768 (578h46m24.758001813s) [upload=507375]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:15:24.886926754 (578h46m24.758009799s) [upload=507375]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:15:25.173177247 (578h46m25.044260282s) [upload=507375]: Done uploading segments... Dec 29 18:15:25.173200431 (578h46m25.044283466s) [upload=507375]: Upload finished. err: <nil> Duration: 30.002740654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 38.33056919s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:15:31 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 38.330596921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.398948358s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:15:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.398974136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.X..X..X..XXXXXX.X.XXX.X....X.X..X.XXX..X.....X.X.....X...] duration: 5.997459877s trace: https://jaeger.dev.storj.io/trace/544cb4babf8ef9ba FAIL (2025-12-29 18:14:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: WK3WM2AIFMTAAAK36TYLRJZPQG5637FND6U6I2FBCIN3RUETQ2VQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: MAAQJF3RSRCTTN327EWULIT7PPZWVZ5JNFUYUQ4ADXWOYLYO6BAA) context deadline exceeded ======LOGS====== ... Dec 29 18:12:20.242385065 (578h43m20.11346811s) [upload=507333]: Got next segment. Inline: false Dec 29 18:12:20.242417347 (578h43m20.113500382s) [upload=507333]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:12:20.242421845 (578h43m20.1135049s) [upload=507333]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:12:20.488510726 (578h43m20.359593771s) [seg_pos={0 0}, upload=507333]: Begin upload segment... Dec 29 18:12:20.488564087 (578h43m20.359647132s) [seg_pos={0 0}, upload=507333]: Done begin upload segment. Dec 29 18:12:20.488599334 (578h43m20.359682369s) [noise=true , address=45.91.116.7:4009 , piece_id=MAAQJF3RSRCTTN327EWULIT7PPZWVZ5JNFUYUQ4ADXWOYLYO6BAA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.489863592 (578h43m20.360946627s) [noise=true , address=77.90.31.13:4626 , piece_id=P5EET5ZR2JMMIN57QOMAGHGWWVPUY53AHNDW5WYSWCSIKV2QIGVQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.490615243 (578h43m20.361698278s) [noise=true , address=145.79.227.47:4266 , piece_id=ANIWW6UUGEYIRRKXCPRRUUKD75HRBTJZOITXDVORG4C2XCLC2KXQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.491185409 (578h43m20.362268454s) [noise=true , address=145.79.227.16:4911 , piece_id=INCXJKLZ35G6XWXAXMOUXFATDAZK555INUA77AJJUP5PUDQIEPFA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.491774901 (578h43m20.362857936s) [noise=true , address=145.79.227.7:4293 , piece_id=O2M2UEHW2JOCUX4DYFVUA45BB32KHR2ULPVJLEESF2TTKCFLK7MA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.492435028 (578h43m20.363518063s) [noise=true , address=145.79.227.40:4051 , piece_id=Y5BBSL4RVNKKMJZTGSWYVR2XDJVQJUXYEHUEVOGYBEFXXSE4FSEQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.494507304 (578h43m20.365590339s) [noise=true , address=145.79.227.13:4804 , piece_id=7XHBFC2PQYIS7P3ADWGQJM6TLHAQ3SDXCBNBT46H2MMWQX22SXUA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.496443060 (578h43m20.367526105s) [noise=true , address=45.91.116.11:4119 , piece_id=5ZSFSS3GPRUEUH7UZSSWZAHE5I2NUYQ3OSQ7FU3TKGHESSI7BQKQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.498317059 (578h43m20.369400104s) [noise=true , address=145.79.227.36:4517 , piece_id=BG3LXHFCT3JZPK5JFV6RTTZHH2LP5FUJMVCOIFWOHIZMFJLEAT6A, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.498879279 (578h43m20.369962314s) [noise=true , address=145.79.227.11:4730 , piece_id=7TNCIUBGJJAONW3WDE5HODZWMDASXGJCEX4APPIC7VQNX3W2C5RQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.500390016 (578h43m20.371473051s) [noise=true , address=145.79.227.8:4329 , piece_id=QOZ73FJZ6YGXFIRJIXG6DD3ZDUTAO2BYZBYCC2BVJ4PQ2SZIB7CQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.501699228 (578h43m20.372782273s) [noise=true , address=145.79.227.42:4123 , piece_id=QSA7YKZ2IHMCKE65K45QPXYKJ5NHYZPQW7SXUOYR2DJPZH5YFYAA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.503409274 (578h43m20.374492309s) [noise=true , address=45.91.116.8:4047 , piece_id=N5RYNF333KI7J4GFYQL562KBGOJ5CQIPYAZEX27BUMH4V6HWVWIA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.503820838 (578h43m20.374903883s) [noise=true , address=145.79.227.15:4874 , piece_id=VBBPYKVNVJHPIDJXI4HOILZYRISTEAGHB5LVJ7KJY6JVKDO4LLAA, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.504171005 (578h43m20.37525405s) [noise=true , address=77.90.31.18:4805 , piece_id=WK3WM2AIFMTAAAK36TYLRJZPQG5637FND6U6I2FBCIN3RUETQ2VQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:20.506018272 (578h43m20.377101317s) [noise=true , address=145.79.227.48:4303 , piece_id=DC4UXRWGKG7F2PWZYCYLKKP6FDTPZDBMLQTS3BVKKG4ACESS2AHQ, seg_pos={0 0}, upload=507333]: Uploading piece... Dec 29 18:12:22.195371085 (578h43m22.06645413s) [upload=507333]: Next returned nil segment Dec 29 18:12:22.195385513 (578h43m22.066468548s) [upload=507333]: Waiting for error group managing segments... Dec 29 18:12:22.467766507 (578h43m22.338849552s) [noise=true , address=145.79.227.36:4517 , piece_id=BG3LXHFCT3JZPK5JFV6RTTZHH2LP5FUJMVCOIFWOHIZMFJLEAT6A, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.482433456 (578h43m22.353516491s) [noise=true , address=145.79.227.15:4874 , piece_id=VBBPYKVNVJHPIDJXI4HOILZYRISTEAGHB5LVJ7KJY6JVKDO4LLAA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.489219366 (578h43m22.360302401s) [noise=true , address=145.79.227.8:4329 , piece_id=QOZ73FJZ6YGXFIRJIXG6DD3ZDUTAO2BYZBYCC2BVJ4PQ2SZIB7CQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.495320223 (578h43m22.366403268s) [noise=true , address=145.79.227.42:4123 , piece_id=QSA7YKZ2IHMCKE65K45QPXYKJ5NHYZPQW7SXUOYR2DJPZH5YFYAA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.503752491 (578h43m22.374835526s) [noise=true , address=145.79.227.7:4293 , piece_id=O2M2UEHW2JOCUX4DYFVUA45BB32KHR2ULPVJLEESF2TTKCFLK7MA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.510736659 (578h43m22.381819694s) [noise=true , address=145.79.227.11:4730 , piece_id=7TNCIUBGJJAONW3WDE5HODZWMDASXGJCEX4APPIC7VQNX3W2C5RQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.644082119 (578h43m22.515165154s) [noise=true , address=145.79.227.13:4804 , piece_id=7XHBFC2PQYIS7P3ADWGQJM6TLHAQ3SDXCBNBT46H2MMWQX22SXUA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.675597579 (578h43m22.546680614s) [noise=true , address=77.90.31.18:4805 , piece_id=WK3WM2AIFMTAAAK36TYLRJZPQG5637FND6U6I2FBCIN3RUETQ2VQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.756459572 (578h43m22.627542607s) [noise=true , address=45.91.116.8:4047 , piece_id=N5RYNF333KI7J4GFYQL562KBGOJ5CQIPYAZEX27BUMH4V6HWVWIA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.764529168 (578h43m22.635612213s) [noise=true , address=45.91.116.11:4119 , piece_id=5ZSFSS3GPRUEUH7UZSSWZAHE5I2NUYQ3OSQ7FU3TKGHESSI7BQKQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.767477903 (578h43m22.638560938s) [noise=true , address=77.90.31.13:4626 , piece_id=P5EET5ZR2JMMIN57QOMAGHGWWVPUY53AHNDW5WYSWCSIKV2QIGVQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.827361962 (578h43m22.698444997s) [noise=true , address=45.91.116.7:4009 , piece_id=MAAQJF3RSRCTTN327EWULIT7PPZWVZ5JNFUYUQ4ADXWOYLYO6BAA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.881446266 (578h43m22.752529301s) [noise=true , address=145.79.227.47:4266 , piece_id=ANIWW6UUGEYIRRKXCPRRUUKD75HRBTJZOITXDVORG4C2XCLC2KXQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: <nil> Dec 29 18:12:22.881465072 (578h43m22.752548107s) [seg_pos={0 0}, upload=507333]: Segment reached optimal threshold of 13 pieces. Dec 29 18:12:22.881705510 (578h43m22.752788555s) [noise=true , address=145.79.227.40:4051 , piece_id=Y5BBSL4RVNKKMJZTGSWYVR2XDJVQJUXYEHUEVOGYBEFXXSE4FSEQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 18:12:22.881712593 (578h43m22.752795628s) [noise=true , address=145.79.227.40:4051 , piece_id=Y5BBSL4RVNKKMJZTGSWYVR2XDJVQJUXYEHUEVOGYBEFXXSE4FSEQ, seg_pos={0 0}, upload=507333]: Long tail cancellation due to optimal threshold Dec 29 18:12:22.881974762 (578h43m22.753057807s) [noise=true , address=145.79.227.16:4911 , piece_id=INCXJKLZ35G6XWXAXMOUXFATDAZK555INUA77AJJUP5PUDQIEPFA, seg_pos={0 0}, upload=507333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: write tcp 188.166.225.170:54950->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54950->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54950->145.79.227.16:4911: use of closed network connection Dec 29 18:12:22.881981015 (578h43m22.75306405s) [noise=true , address=145.79.227.16:4911 , piece_id=INCXJKLZ35G6XWXAXMOUXFATDAZK555INUA77AJJUP5PUDQIEPFA, seg_pos={0 0}, upload=507333]: Long tail cancellation due to optimal threshold Dec 29 18:12:22.882017113 (578h43m22.753100148s) [noise=true , address=145.79.227.48:4303 , piece_id=DC4UXRWGKG7F2PWZYCYLKKP6FDTPZDBMLQTS3BVKKG4ACESS2AHQ, seg_pos={0 0}, upload=507333]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:56190->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:56190->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56190->145.79.227.48:4303: use of closed network connection Dec 29 18:12:22.882025579 (578h43m22.753108614s) [noise=true , address=145.79.227.48:4303 , piece_id=DC4UXRWGKG7F2PWZYCYLKKP6FDTPZDBMLQTS3BVKKG4ACESS2AHQ, seg_pos={0 0}, upload=507333]: Long tail cancellation due to optimal threshold Dec 29 18:12:22.882079582 (578h43m22.753162617s) [seg_pos={0 0}, upload=507333]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:12:22.882161699 (578h43m22.753244734s) [upload=507333]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:12:22.882167209 (578h43m22.753250244s) [upload=507333]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:12:23.198380756 (578h43m23.069463791s) [upload=507333]: Done uploading segments... Dec 29 18:12:23.198400665 (578h43m23.0694837s) [upload=507333]: Upload finished. err: <nil> Duration: 5.997478713s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)