Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.447857818s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:59:19 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.447887985s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X.....X...X.......X.........X.............X............X....] duration: 4.986784667s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 12:59:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.986817481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...................X..............XXXX......................] duration: 8.113310924s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:59:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.113338577s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......X...X...X..XX.........XXX................X............] duration: 10.990932248s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:59:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.990958257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.182691866s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:58:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.18271472s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.....X...X.......X.........X.............X............X...] duration: 4.787112807s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 12:58:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.787146641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 39.106902624s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:58:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 39.106929356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......X...X...X..XX.........XXX................X...........] duration: 6.927766684s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:58:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.927793726s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....................X..............XXXX.....................] duration: 16.4049857s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:58:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.405015176s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [........X...X...X..XX.........XXX................X..........] duration: 8.994495393s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:58:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.994529778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.........X...X...X..XX.........XXX................X.........] duration: 5.780189628s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:57:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.780220276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.524165026s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:57:51 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.524206274s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 16.59748622s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:57:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.597518422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.....X...X.......X.........X.............X............X..] duration: 3.699441075s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 12:57:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.699496001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................X..............XXXX....................] duration: 15.368473609s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:57:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.368508556s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.311947992s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:57:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.311976505s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 23.205892191s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:57:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 23.205923841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X..............XXXX...................] duration: 9.510685277s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:57:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.51071312s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..........X...X...X..XX.........XXX................X........] duration: 9.570364369s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:56:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.570392092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.279015841s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:56:44 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.279075965s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.....X...X.......X.........X.............X............X.] duration: 5.000348999s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 12:56:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000446694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 24.68927762s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:56:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 24.68930363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.739405488s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:56:29 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.739506079s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.126699826s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:56:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.126734785s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................X..............XXXX..................] duration: 9.777826111s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:56:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.777861068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X.....X...X.......X.........X.............X............X] duration: 5.000609466s trace: https://jaeger.dev.storj.io/trace/57cbebd4f16c5c7c FAIL (2025-12-09 12:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000635845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...........X...X...X..XX.........XXX................X.......] duration: 10.500389354s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:55:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.50042443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 24.644002398s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:55:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 24.644053706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.493262511s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:55:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.493294162s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................X..............XXXX.................] duration: 11.054505609s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:55:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 11.054540485s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X.....X...X.......X.........X.............X............] duration: 3.841791784s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:55:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.841821271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................X..............XXXX................] duration: 9.6347601s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:54:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.634789988s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.704862308s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:54:45 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.70488458s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [............X...X...X..XX.........XXX................X......] duration: 6.47153759s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:54:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.471555645s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X.....X...X.......X.........X.............X...........] duration: 4.322844837s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:54:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.322887057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 17.827273351s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:54:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 17.827317115s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................X..............XXXX...............] duration: 10.517640699s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:54:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.51767299s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.28607665s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:54:08 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.28616569s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.............X...X...X..XX.........XXX................X.....] duration: 6.715207047s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:54:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.715237253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 4.992466346s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:53:45 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 4.992582267s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 20.091871426s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:53:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 20.091903718s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X.....X...X.......X.........X.............X..........] duration: 3.772513232s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:53:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.772550673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..............X...X...X..XX.........XXX................X....] duration: 7.880776521s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:53:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.880809926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X..............XXXX..............] duration: 9.520098375s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:52:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.520194296s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.553857053s trace: https://jaeger.dev.storj.io/trace/6a91d5ff6f647fca FAIL (2025-12-09 12:52:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 5.553887713s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 16.647924634s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:52:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.647951215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X.....X...X.......X.........X.............X.........] duration: 4.173759947s trace: https://jaeger.dev.storj.io/trace/5383613973694fca FAIL (2025-12-09 12:52:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.173789683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X..............XXXX.............] duration: 9.440601354s trace: https://jaeger.dev.storj.io/trace/5fe208e29818b161 FAIL (2025-12-09 12:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Dec 9 12:41:21.031438079 (93h12m20.902521114s) [upload=81270]: Starting upload Dec 9 12:41:21.031494307 (93h12m20.902577342s) [upload=81270]: Done starting upload Dec 9 12:41:21.031672055 (93h12m20.9027551s) [upload=81270]: Uploading segments... Dec 9 12:41:21.031736478 (93h12m20.902819513s) [upload=81270]: Got next segment. Inline: false Dec 9 12:41:21.031763729 (93h12m20.902846774s) [upload=81270]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:41:21.031768629 (93h12m20.902851674s) [upload=81270]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:41:21.376247446 (93h12m21.247330482s) [seg_pos={0 0}, upload=81270]: Begin upload segment... Dec 9 12:41:21.376296279 (93h12m21.247379314s) [seg_pos={0 0}, upload=81270]: Done begin upload segment. Dec 9 12:41:21.376320125 (93h12m21.24740317s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376335403 (93h12m21.247418439s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376705569 (93h12m21.247788604s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.376991854 (93h12m21.248074889s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377271847 (93h12m21.248354882s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.377783762 (93h12m21.248866807s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378150800 (93h12m21.249233845s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378581341 (93h12m21.249664386s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.378935825 (93h12m21.25001886s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.379271956 (93h12m21.250354991s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.380485255 (93h12m21.2515683s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.381652639 (93h12m21.252735674s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.383075969 (93h12m21.254159004s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.384176855 (93h12m21.2552599s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.385441093 (93h12m21.256524138s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:21.386756297 (93h12m21.257839342s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Uploading piece... Dec 9 12:41:22.442658511 (93h12m22.313741556s) [noise=true , address=77.90.31.13:4628 , piece_id=2N44IAPV5Z3PADNR6ZCJQLXXER6UW5UIY2W62VVNPGTEG4U3SCUQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.486279844 (93h12m22.357362899s) [noise=true , address=77.90.31.12:4592 , piece_id=J4KVW2SN2MFY4NICA53KDIZDRFWKM5TNVPBSPMK6B27W4YCLA2LQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:22.494863800 (93h12m22.365946835s) [noise=true , address=77.90.31.13:4627 , piece_id=ELC2HR3UGSNPDMY6ZI5BEDE6CQOAN2LBWAIODPJ3MOMPIHLJCSOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: storage node overloaded, request limit: 2000 Dec 9 12:41:23.209374854 (93h12m23.080457899s) [upload=81270]: Next returned nil segment Dec 9 12:41:23.209528616 (93h12m23.080611661s) [upload=81270]: Waiting for error group managing segments... Dec 9 12:41:23.560314774 (93h12m23.431397819s) [noise=true , address=77.90.31.11:4553 , piece_id=6K7H3ZEHEWYQNCOXYJCR6UCIMTNV6YBQ6K2SIKEMTV3IVZ2VHJQQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:23.615677957 (93h12m23.486761002s) [noise=true , address=77.90.31.18:4807 , piece_id=TNDUJISQXDHFNV52JLUUVY2VELESF3GS7CLNWNXZIVS7NAG2IAVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238335694 (93h12m24.109418729s) [noise=true , address=145.79.227.32:4835 , piece_id=6WOXP4INOQIJZU7DYWLO4MYBYZL5XZMUTWVWA4ZITAQ5AHEXC7YQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.238615988 (93h12m24.109699023s) [noise=true , address=145.79.227.24:4583 , piece_id=GIJVVBQR4JGHOEQ54NLVKMIA2R6EU7AX4KCBAL3JDDEI5UMEWVYA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.255953782 (93h12m24.127036827s) [noise=true , address=145.79.227.42:4121 , piece_id=PARBS64M4B6I5VX553UKD7ZLFE6G2HBM72DVIHOZ6Q3ALXIZ7H6Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.391697325 (93h12m24.26278036s) [noise=true , address=145.79.227.41:4086 , piece_id=CBG4KXL6J3FL3A52XXEBULXVYCZWB5EF7ADD5JUESQFL7XJZY6NQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.402491670 (93h12m24.273574715s) [noise=true , address=145.79.227.24:4581 , piece_id=I7ZHGGF5ZYN2TJRBVIZ2YAVLXCSM53XSCLECAIVFKV5MKKIFQ46A, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.443519245 (93h12m24.31460229s) [noise=true , address=145.79.227.26:4655 , piece_id=SVVDYWC6LPK3GAHMIV54QMHT4T5V2WQGK2JQKEDVAVQS4RIGJTRA, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.497956865 (93h12m24.3690399s) [noise=true , address=145.79.227.23:4546 , piece_id=RV6EVPGF2HEMWU4EYNXF6W2IGYGLK5M2B3A4GG2W2IXXR665QFVQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510225167 (93h12m24.381308212s) [noise=true , address=145.79.227.40:4050 , piece_id=ZTI7IY7U5IRWFCHDH5FQ7JZZEZOOCAAW4SYGNKKQS66U47FG2X7Q, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:24.510378670 (93h12m24.381461705s) [noise=true , address=145.79.227.21:4473 , piece_id=FU7O3TDJSKWS77IV6UVK6FBOSPDPFIMUXQI4SCPKSV7U7TXPQ2WQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: <nil> Dec 9 12:41:50.063931035 (93h12m49.935014071s) [noise=true , address=77.90.31.16:4735 , piece_id=YZN44IFHWZJGNSKWIWWRSUXGJMAWF27CJ4UEV6T3Y4NEQSFGZ2XQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064136978 (93h12m49.935220023s) [noise=true , address=145.79.227.38:4977 , piece_id=OLG4PECZ3EOXC3UON25UIRNEKRJ5SLDAYQEKNJRTVDKL7S7JAXOQ, seg_pos={0 0}, upload=81270]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: expected piece hash; context deadline exceeded; EOF Dec 9 12:41:50.064165783 (93h12m49.935248828s) [seg_pos={0 0}, upload=81270]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Dec 9 12:41:50.417440641 (93h12m50.288523696s) [upload=81270]: Done uploading segments... Dec 9 12:41:50.417479364 (93h12m50.288562399s) [upload=81270]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.440633224s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...............X...X...X..XX.........XXX................X...] duration: 8.91818793s trace: https://jaeger.dev.storj.io/trace/7c34418af454e9fe FAIL (2025-12-09 12:51:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.918216304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 9.349121663s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:51:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.349138304s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)