Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X..............XXXX...............................] duration: 9.675290019s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:06:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.675313604s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.326152527s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:06:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.326184216s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X.........X.............X............X.......XX.XX...] duration: 4.952931953s trace: https://jaeger.dev.storj.io/trace/43e5ea3b1f9b8d99 FAIL (2025-12-09 13:06:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.952960778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.148971855s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:06:15 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.149007524s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 29.572033964s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:06:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 29.572064573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.968243901s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:05:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.968271212s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.X...X..XX.........XXX................X.....................] duration: 6.181925446s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:05:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.181955274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 9.303722394s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:05:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.303743565s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.......X.........X.............X............X.......XX.XX..] duration: 4.947441962s trace: https://jaeger.dev.storj.io/trace/43e5ea3b1f9b8d99 FAIL (2025-12-09 13:05:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.947465057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 2m15.791253774s trace: https://jaeger.dev.storj.io/trace/435b3249ab0873c2 FAIL (2025-12-09 13:05:33 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2m15.791277007s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X..............XXXX..............................] duration: 11.165723731s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:05:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.165752406s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..X...X..XX.........XXX................X....................] duration: 8.466236467s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:05:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.466294557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.......X.........X.............X............X.......XX.XX.] duration: 4.702780238s trace: https://jaeger.dev.storj.io/trace/43e5ea3b1f9b8d99 FAIL (2025-12-09 13:04:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.702811076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.58500504s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:04:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.585029406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.228815609s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:04:48 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.228844394s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X..............XXXX.............................] duration: 9.809702892s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:04:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.809731877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 15.723856129s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:04:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.723883982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X..............XXXX............................] duration: 10.169764146s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:04:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.169793049s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X...X..XX.........XXX................X...................] duration: 6.991674069s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:04:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.991700298s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.917800546s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:03:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.917829201s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.......X.........X.............X............X.......XX.XX] duration: 5.000501289s trace: https://jaeger.dev.storj.io/trace/43e5ea3b1f9b8d99 FAIL (2025-12-09 13:03:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000527558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 35.973390088s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:03:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 35.973415627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X.......X.........X.............X............X.......XX.X] duration: 5.001367486s trace: https://jaeger.dev.storj.io/trace/d8a65ed9d6c618d FAIL (2025-12-09 13:03:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001387183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X...X...X..XX.........XXX................X..................] duration: 9.760805951s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:03:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.760839245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X..............XXXX...........................] duration: 10.450847843s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:03:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.45087738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X...X.......X.........X.............X............X.......XX.] duration: 4.97615117s trace: https://jaeger.dev.storj.io/trace/3676b8ec61a964e0 FAIL (2025-12-09 13:03:07 UTC) orbiter warning: context deadline exceeded Duration: 4.976178943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.299487249s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:03:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.299557303s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X...X.......X.........X.............X............X.......XX] duration: 5.001272795s trace: https://jaeger.dev.storj.io/trace/3676b8ec61a964e0 FAIL (2025-12-09 13:02:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001294706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 9.495832743s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:02:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.495861558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...X...X..XX.........XXX................X.................] duration: 12.907309558s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:02:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 12.90734161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.464311893s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:02:10 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.464345496s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X..............XXXX..........................] duration: 9.549418784s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.549470231s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...X...X..XX.........XXX................X................] duration: 9.360597123s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:01:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.360626719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 23.786607238s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:01:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 23.786756593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.383229041s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:01:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.383260881s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X..............XXXX.........................] duration: 14.030951776s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.03098068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X...X.......X.........X.............X............X.......X] duration: 5.001541277s trace: https://jaeger.dev.storj.io/trace/6a3b8b4608a1d57c FAIL (2025-12-09 13:01:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001674741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 27.055195062s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:01:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 27.055219177s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...X...X.......X.........X.............X............X.......] duration: 3.734266157s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 13:01:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.734297378s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...X...X..XX.........XXX................X...............] duration: 8.998634472s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 13:01:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.998665461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X..............XXXX........................] duration: 9.780775496s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.780806735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 40.443492171s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 13:00:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 40.443705026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.507654443s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:00:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.507699168s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X...X.......X.........X.............X............X......] duration: 4.915518838s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 13:00:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.915543725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X..............XXXX.......................] duration: 14.171510563s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 13:00:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 14.171551872s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.550584281s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 13:00:01 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.550618085s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X...X...X..XX.........XXX................X..............] duration: 9.25013433s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:59:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.250160661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X...X.......X.........X.............X............X.....] duration: 5.001460889s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 12:59:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001479454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....X...X...X..XX.........XXX................X.............] duration: 9.263536802s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:59:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.263561428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 15.725547718s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:59:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.725580489s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)