Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X....................X....X...........] duration: 5.803211176s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:38:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.803236184s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....................X....X..........] duration: 5.834352095s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:37:46 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.834378777s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.392386353s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:37:19 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.392415657s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.628738529s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:37:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.628789286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.452850235s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:37:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.452890572s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.......X.............XX.X.........X.....................] duration: 5.863037217s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:36:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.863062224s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....................X....X.........] duration: 5.889725284s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:36:44 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.889745983s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.510545969s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.510572058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.396942598s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:36:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.396958508s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.249851363s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:36:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.249878737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....................X....X........] duration: 5.820839072s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:36:14 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.820867678s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....................X....X.......] duration: 5.682192105s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:35:58 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.682222203s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.554342265s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:35:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.554388774s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X.......X.............XX.X.........X....................] duration: 5.996289089s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:35:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.996314909s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.500327373s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:35:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500451669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000649668s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 15:35:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000662814s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X.......X.............XX.X.........X...................] duration: 6.060358058s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:34:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 6.060382995s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....................X....X......] duration: 11.815032821s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:34:55 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 11.815061667s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.441446212s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.44147111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.......X.............XX.X.........X..................] duration: 5.672385261s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:34:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.672424606s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.731861221s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:34:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.731889395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....................X....X.....] duration: 6.34649884s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:34:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.34657779s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.......X.............XX.X.........X.................] duration: 5.705645025s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:33:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.7056847s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.559099094s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:33:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559127466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.......X.............XX.X.........X................] duration: 5.765825469s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:33:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.765853562s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....................X....X....] duration: 5.384723405s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:32:59 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.384748552s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.546985507s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:32:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547011327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.......X.............XX.X.........X...............] duration: 5.68088181s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:32:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.6809078s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....................X....X...] duration: 5.458127588s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:31:54 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.458153217s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.......X.............XX.X.........X..............] duration: 5.987980248s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:31:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.988007829s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.481497857s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:31:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481523407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....................X....X..] duration: 5.576827213s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:31:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.576850376s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.735202825s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:30:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.735224035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.......X.............XX.X.........X.............] duration: 5.696010669s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.696041417s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.57956309s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:30:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.579594851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....................X....X.] duration: 5.4333041s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:29:58 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.433343064s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.646005577s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:29:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646032589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.......X.............XX.X.........X............] duration: 5.856915199s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:29:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.856947991s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.359282217s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:29:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.359330571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X....................X....X] duration: 30.002154328s trace: https://jaeger.dev.storj.io/trace/231ad5e6801d138a FAIL (2025-12-23 15:29:09 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.002177733s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.......X.............XX.X.........X...........] duration: 5.692260467s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:28:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.692286357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 4.764531264s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:28:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.764779386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.739121859s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:28:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.739151545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X....................X....] duration: 5.524053592s trace: https://jaeger.dev.storj.io/trace/150a5c56431f829 FAIL (2025-12-23 15:28:07 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.524076287s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.......X.............XX.X.........X..........] duration: 5.661883806s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:28:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.661916367s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X....................X...] duration: 5.168111588s trace: https://jaeger.dev.storj.io/trace/150a5c56431f829 FAIL (2025-12-23 15:27:50 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.168143169s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X.......X.............XX.X.........X.........] duration: 11.678249476s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:27:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 11.678275246s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.479560299s trace: https://jaeger.dev.storj.io/trace/24ee53b3af18e597 FAIL (2025-12-23 15:27:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479585476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X.......X.............XX.X.........X........] duration: 5.702992421s trace: https://jaeger.dev.storj.io/trace/237db41ff0ec7678 FAIL (2025-12-23 15:27:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ) context deadline exceeded ======LOGS====== 23 15:22:16.238657063 (431h53m16.109740108s) [upload=379126]: Starting upload Dec 23 15:22:16.238765670 (431h53m16.109848705s) [upload=379126]: Done starting upload Dec 23 15:22:16.238882502 (431h53m16.109965537s) [upload=379126]: Uploading segments... Dec 23 15:22:16.239047566 (431h53m16.110130611s) [upload=379126]: Got next segment. Inline: false Dec 23 15:22:16.239074798 (431h53m16.110157833s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 15:22:16.239079126 (431h53m16.110162161s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 15:22:16.493362515 (431h53m16.36444556s) [seg_pos={0 0}, upload=379126]: Begin upload segment... Dec 23 15:22:16.493406800 (431h53m16.364489845s) [seg_pos={0 0}, upload=379126]: Done begin upload segment. Dec 23 15:22:16.493436466 (431h53m16.364519501s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.493508483 (431h53m16.364591518s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494181154 (431h53m16.365264189s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494562541 (431h53m16.365645576s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.494985736 (431h53m16.366068771s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495342546 (431h53m16.366425591s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.495681211 (431h53m16.366764256s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.496002983 (431h53m16.367086018s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.497027925 (431h53m16.36811097s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.498334433 (431h53m16.369417468s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.499499351 (431h53m16.370582386s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.500552677 (431h53m16.371635722s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.501752070 (431h53m16.372835105s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.503228432 (431h53m16.374311477s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.504119428 (431h53m16.375202473s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:16.505431046 (431h53m16.376514091s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Uploading piece... Dec 23 15:22:18.148832013 (431h53m18.019915058s) [upload=379126]: Next returned nil segment Dec 23 15:22:18.148845448 (431h53m18.019928483s) [upload=379126]: Waiting for error group managing segments... Dec 23 15:22:18.472932646 (431h53m18.344015681s) [noise=true , address=145.79.227.11:4729 , piece_id=LOOTN7YEG42QF5MCM3J2SFMZABBIDN6DDPKCXWGI3F5BLOFATJGA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.473182241 (431h53m18.344265286s) [noise=true , address=145.79.227.46:4601 , piece_id=OGQ6BVIKG37VWKTBUI2OF2MQAAOBT4L5OQIXHXDM3PRG3MILUIWA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.474241748 (431h53m18.345324793s) [noise=true , address=145.79.227.16:4910 , piece_id=TRZSNP6VDSGLWJE7NYP7BAF4OC3F34WA5OUF42M72AAAEK6RKSVA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.487309662 (431h53m18.358392697s) [noise=true , address=145.79.227.21:4473 , piece_id=5JFWCYU6KTXK424RLSOVIL3SX4U4ASEDTANBLMJJIYFURK2JK2DA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.488424665 (431h53m18.3595077s) [noise=true , address=145.79.227.29:4942 , piece_id=EV5E7CENXPKHFVX2TUM7RTLEINEZQN3NEA56YQSLECLSJ3PVABJQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.493788940 (431h53m18.364871975s) [noise=true , address=145.79.227.49:4339 , piece_id=HW4NH7E6ZP3NQTS5FSTB2YGQQT3I2PT5QE55MTJGDMG75O2AQJRQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.498076363 (431h53m18.369159398s) [noise=true , address=145.79.227.17:4970 , piece_id=RNU342WQPZLMGYGAGZ5WGGPFMPATUUEEGVPAHXFOXPGT6A4P6GTQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.507748471 (431h53m18.378831516s) [noise=true , address=145.79.227.13:4802 , piece_id=3XWSRWZEOD5DUEUCWI27EESAMKEYX2CB2WEEJIXZ4QHSJRJIFHBQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.509533971 (431h53m18.380617016s) [noise=true , address=145.79.227.39:4014 , piece_id=L4AILEYDBBVK4XHLHWCEIJT6BOCZEUZYONBUL756CCNWJC72AUTA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.515264242 (431h53m18.386347287s) [noise=true , address=145.79.227.37:4565 , piece_id=I3OOHUIU7E3S5OHTBZRWMK35AFAYUK72BPPHDMNPDFKZ2XV4ZLXQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.541493156 (431h53m18.412576191s) [noise=true , address=145.79.227.6:4259 , piece_id=5FCH36ETGSFWVERCXHMGNMCYKANXNPJWAM5UHONKYYA7SFID6GUQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.777340640 (431h53m18.648423686s) [noise=true , address=45.91.116.13:4156 , piece_id=4ELDWWD6EZ4H4C4PXWIN67VGXQVPU2NUAM3OMQFTOEBQAFNHFIKQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779642984 (431h53m18.650726019s) [noise=true , address=45.91.116.12:4225 , piece_id=PZDL77D6XHAA7GKUB76IQX2JYNP43QBJ3ZDDU46AOVPWYG7V2FNA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: <nil> Dec 23 15:22:18.779656199 (431h53m18.650739244s) [seg_pos={0 0}, upload=379126]: Segment reached optimal threshold of 13 pieces. Dec 23 15:22:18.779871559 (431h53m18.650954614s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.779882721 (431h53m18.650965766s) [noise=true , address=45.91.116.9:4262 , piece_id=R7HW4CZ5OEJQCRJPYB7UXOB2FPP6VVJNWHBWSLOLABAUO3ZJKKDQ, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780051421 (431h53m18.651134456s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50886->45.91.116.5:4300: use of closed network connection Dec 23 15:22:18.780083132 (431h53m18.651166167s) [noise=true , address=45.91.116.5:4300 , piece_id=XPDRM6Y6XOKVQRJ4YPOOC63IVD4D2J67SZIX7ZDLJ4GYKPAXPA3A, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780113520 (431h53m18.651196565s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 23 15:22:18.780119321 (431h53m18.651202356s) [noise=true , address=45.91.116.10:4082 , piece_id=TWGVS6DSDHAWKASJFKRSA5EDOMX6EONXLDE3IPS7QUKAOH4MAGQA, seg_pos={0 0}, upload=379126]: Long tail cancellation due to optimal threshold Dec 23 15:22:18.780135041 (431h53m18.651218076s) [seg_pos={0 0}, upload=379126]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 15:22:18.780249248 (431h53m18.651332293s) [upload=379126]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 15:22:18.780254588 (431h53m18.651337633s) [upload=379126]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 15:22:19.067496005 (431h53m18.93857905s) [upload=379126]: Done uploading segments... Dec 23 15:22:19.067523939 (431h53m18.938606984s) [upload=379126]: Upload finished. err: <nil> Duration: 5.703019234s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....................X..] duration: 5.200047319s trace: https://jaeger.dev.storj.io/trace/150a5c56431f829 FAIL (2025-12-23 15:26:53 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.200078649s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)