Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X........X......................] duration: 38.797332308s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:44:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 38.797414384s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXXX] duration: 5.001194719s trace: https://jaeger.dev.storj.io/trace/1521a57455a452aa FAIL (2025-12-28 13:44:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001217492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XX....X...........X....................X............X.......] duration: 20.496997388s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:44:30 UTC) orbiter warning: context deadline exceeded Duration: 20.497028085s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 10.496804883s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:44:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 10.496837256s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.XX..XX..X.......XXX......XX......X..X.....X..X.X...XX..] duration: 5.687988485s trace: https://jaeger.dev.storj.io/trace/94c6dc4e5f68a6e FAIL (2025-12-28 13:44:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA) context deadline exceeded ======LOGS====== 550h13m27.258022633s) [upload=482419]: Done starting upload Dec 28 13:42:27.387099462 (550h13m27.258182497s) [upload=482419]: Uploading segments... Dec 28 13:42:27.387357324 (550h13m27.258440459s) [upload=482419]: Got next segment. Inline: false Dec 28 13:42:27.387408080 (550h13m27.258491115s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:42:27.387413841 (550h13m27.258496886s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:42:27.638563682 (550h13m27.509646727s) [seg_pos={0 0}, upload=482419]: Begin upload segment... Dec 28 13:42:27.638646689 (550h13m27.509729724s) [seg_pos={0 0}, upload=482419]: Done begin upload segment. Dec 28 13:42:27.638711563 (550h13m27.509794608s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.638725659 (550h13m27.509808705s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.639592059 (550h13m27.510675094s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640287844 (550h13m27.511370879s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640759092 (550h13m27.511842137s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.641490505 (550h13m27.51257354s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.642874049 (550h13m27.513957084s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.643700002 (550h13m27.514783047s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.644468685 (550h13m27.51555173s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.645884662 (550h13m27.516967697s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.647456905 (550h13m27.518539941s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648041389 (550h13m27.519124424s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648974235 (550h13m27.52005727s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.650328172 (550h13m27.521411217s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.651182510 (550h13m27.522265545s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.653389852 (550h13m27.524472887s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:29.259891664 (550h13m29.130974699s) [upload=482419]: Next returned nil segment Dec 28 13:42:29.259904918 (550h13m29.130987954s) [upload=482419]: Waiting for error group managing segments... Dec 28 13:42:29.646472706 (550h13m29.517555751s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.649315019 (550h13m29.520398064s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655433360 (550h13m29.526516395s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655570040 (550h13m29.526653075s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.656816143 (550h13m29.527899178s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.662269416 (550h13m29.533352461s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.664572742 (550h13m29.535655787s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.678652472 (550h13m29.549735517s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.683770548 (550h13m29.554853584s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684176270 (550h13m29.555259315s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684578737 (550h13m29.555661772s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.901472728 (550h13m29.772555773s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905337859 (550h13m29.776420894s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905361523 (550h13m29.776444568s) [seg_pos={0 0}, upload=482419]: Segment reached optimal threshold of 13 pieces. Dec 28 13:42:29.905586202 (550h13m29.776669237s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection Dec 28 13:42:29.905593856 (550h13m29.776676891s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.905731959 (550h13m29.776814994s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 13:42:29.905738521 (550h13m29.776821556s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906284732 (550h13m29.777367777s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:58736->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:42:29.906304299 (550h13m29.777387344s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906332723 (550h13m29.777415768s) [seg_pos={0 0}, upload=482419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:42:29.906459084 (550h13m29.777542129s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:42:29.906466838 (550h13m29.777549883s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:42:30.209359044 (550h13m30.080442089s) [upload=482419]: Done uploading segments... Dec 28 13:42:30.209387108 (550h13m30.080470143s) [upload=482419]: Upload finished. err: <nil> Duration: 5.688024865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X........X.....................] duration: 43.817713449s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:43:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 43.817732965s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.XX....X...........X....................X............X......] duration: 17.374002134s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:43:53 UTC) orbiter warning: context deadline exceeded Duration: 17.374030216s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 51 out of 60 is failed (threshold=6) [X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXXX] duration: 5.000935935s trace: https://jaeger.dev.storj.io/trace/3c030f5287ac5653 FAIL (2025-12-28 13:43:29 UTC) orbiter warning: context deadline exceeded Duration: 5.000957095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 7.993097215s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:43:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 7.993126421s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [....X.XX..XX..X.......XXX......XX......X..X.....X..X.X...XX.] duration: 6.163433262s trace: https://jaeger.dev.storj.io/trace/94c6dc4e5f68a6e FAIL (2025-12-28 13:43:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA) context deadline exceeded ======LOGS====== 550h13m27.258022633s) [upload=482419]: Done starting upload Dec 28 13:42:27.387099462 (550h13m27.258182497s) [upload=482419]: Uploading segments... Dec 28 13:42:27.387357324 (550h13m27.258440459s) [upload=482419]: Got next segment. Inline: false Dec 28 13:42:27.387408080 (550h13m27.258491115s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:42:27.387413841 (550h13m27.258496886s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:42:27.638563682 (550h13m27.509646727s) [seg_pos={0 0}, upload=482419]: Begin upload segment... Dec 28 13:42:27.638646689 (550h13m27.509729724s) [seg_pos={0 0}, upload=482419]: Done begin upload segment. Dec 28 13:42:27.638711563 (550h13m27.509794608s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.638725659 (550h13m27.509808705s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.639592059 (550h13m27.510675094s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640287844 (550h13m27.511370879s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640759092 (550h13m27.511842137s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.641490505 (550h13m27.51257354s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.642874049 (550h13m27.513957084s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.643700002 (550h13m27.514783047s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.644468685 (550h13m27.51555173s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.645884662 (550h13m27.516967697s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.647456905 (550h13m27.518539941s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648041389 (550h13m27.519124424s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648974235 (550h13m27.52005727s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.650328172 (550h13m27.521411217s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.651182510 (550h13m27.522265545s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.653389852 (550h13m27.524472887s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:29.259891664 (550h13m29.130974699s) [upload=482419]: Next returned nil segment Dec 28 13:42:29.259904918 (550h13m29.130987954s) [upload=482419]: Waiting for error group managing segments... Dec 28 13:42:29.646472706 (550h13m29.517555751s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.649315019 (550h13m29.520398064s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655433360 (550h13m29.526516395s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655570040 (550h13m29.526653075s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.656816143 (550h13m29.527899178s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.662269416 (550h13m29.533352461s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.664572742 (550h13m29.535655787s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.678652472 (550h13m29.549735517s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.683770548 (550h13m29.554853584s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684176270 (550h13m29.555259315s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684578737 (550h13m29.555661772s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.901472728 (550h13m29.772555773s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905337859 (550h13m29.776420894s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905361523 (550h13m29.776444568s) [seg_pos={0 0}, upload=482419]: Segment reached optimal threshold of 13 pieces. Dec 28 13:42:29.905586202 (550h13m29.776669237s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection Dec 28 13:42:29.905593856 (550h13m29.776676891s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.905731959 (550h13m29.776814994s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 13:42:29.905738521 (550h13m29.776821556s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906284732 (550h13m29.777367777s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:58736->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:42:29.906304299 (550h13m29.777387344s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906332723 (550h13m29.777415768s) [seg_pos={0 0}, upload=482419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:42:29.906459084 (550h13m29.777542129s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:42:29.906466838 (550h13m29.777549883s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:42:30.209359044 (550h13m30.080442089s) [upload=482419]: Done uploading segments... Dec 28 13:42:30.209387108 (550h13m30.080470143s) [upload=482419]: Upload finished. err: <nil> Duration: 6.163460604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X........X....................] duration: 50.376702495s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:43:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 50.376727341s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXXX] duration: 5.001507755s trace: https://jaeger.dev.storj.io/trace/71a4c4258f40fe11 FAIL (2025-12-28 13:43:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001528483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 8.623253066s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:42:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 8.623277401s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.....X.XX..XX..X.......XXX......XX......X..X.....X..X.X...XX] duration: 30.004359716s trace: https://jaeger.dev.storj.io/trace/94c6dc4e5f68a6e FAIL (2025-12-28 13:42:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA) context deadline exceeded ======LOGS====== 550h13m27.258022633s) [upload=482419]: Done starting upload Dec 28 13:42:27.387099462 (550h13m27.258182497s) [upload=482419]: Uploading segments... Dec 28 13:42:27.387357324 (550h13m27.258440459s) [upload=482419]: Got next segment. Inline: false Dec 28 13:42:27.387408080 (550h13m27.258491115s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:42:27.387413841 (550h13m27.258496886s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:42:27.638563682 (550h13m27.509646727s) [seg_pos={0 0}, upload=482419]: Begin upload segment... Dec 28 13:42:27.638646689 (550h13m27.509729724s) [seg_pos={0 0}, upload=482419]: Done begin upload segment. Dec 28 13:42:27.638711563 (550h13m27.509794608s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.638725659 (550h13m27.509808705s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.639592059 (550h13m27.510675094s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640287844 (550h13m27.511370879s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.640759092 (550h13m27.511842137s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.641490505 (550h13m27.51257354s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.642874049 (550h13m27.513957084s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.643700002 (550h13m27.514783047s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.644468685 (550h13m27.51555173s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.645884662 (550h13m27.516967697s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.647456905 (550h13m27.518539941s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648041389 (550h13m27.519124424s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.648974235 (550h13m27.52005727s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.650328172 (550h13m27.521411217s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.651182510 (550h13m27.522265545s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:27.653389852 (550h13m27.524472887s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Uploading piece... Dec 28 13:42:29.259891664 (550h13m29.130974699s) [upload=482419]: Next returned nil segment Dec 28 13:42:29.259904918 (550h13m29.130987954s) [upload=482419]: Waiting for error group managing segments... Dec 28 13:42:29.646472706 (550h13m29.517555751s) [noise=true , address=145.79.227.14:4840 , piece_id=G43RQEQKTDZP6TB2B2QWEV7RU3GCPHJIPGV7LUO74XZ5HVH5EMLQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.649315019 (550h13m29.520398064s) [noise=true , address=145.79.227.4:4185 , piece_id=NRWPOGHGMGONL73OWQJVSSUW4J55JOMPRCTTV7XT4T6PUAXUN5FA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655433360 (550h13m29.526516395s) [noise=true , address=145.79.227.50:4375 , piece_id=Y4JMIV5P7WFXBVHAR3TFCRPXBQXKMI6MMWP4MB4IHFYXVMCLTCHQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.655570040 (550h13m29.526653075s) [noise=true , address=145.79.227.19:4077 , piece_id=NHUEYSUGQI74SD2BRF66JYO2YI43RWCDP32AEIPQ6UHJITDRSV7Q, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.656816143 (550h13m29.527899178s) [noise=true , address=145.79.227.12:4765 , piece_id=NKDNKDIFZILC7WZ6TA6PLDOUBI5B5MSC5ODIFZ3GBY32NCL2XXTQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.662269416 (550h13m29.533352461s) [noise=true , address=145.79.227.2:4113 , piece_id=76UVH2VEX2IFFPOSKWETT7QHWY3N7UYRXTQXTLTY5TR67SCHKHZA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.664572742 (550h13m29.535655787s) [noise=true , address=145.79.227.22:4511 , piece_id=YVYVRCHQEQSH2ZBEQAHVC2VQLVAWKVYA4PDDV23AIQ5ZL6TTR5GA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.678652472 (550h13m29.549735517s) [noise=true , address=145.79.227.16:4909 , piece_id=G57S4ISUV2SLVHM5XCUBTL7W6GURK243BAR2J2XAJP3LNEENWHQA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.683770548 (550h13m29.554853584s) [noise=true , address=145.79.227.39:4014 , piece_id=4ZZHXHZ6SHML365DBCMLKYEKGVIKCW2AX6UMTGI6RLIHLI6R6DEQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684176270 (550h13m29.555259315s) [noise=true , address=145.79.227.48:4303 , piece_id=EIDEUTMM2EQLMCTUROIO3QT27S35W7DCZLRB3RLVPTYY2Y5WTBUQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.684578737 (550h13m29.555661772s) [noise=true , address=145.79.227.38:4979 , piece_id=RTCWALUQO5USYX7UW556SHA4AL2336LTT5NAAGZ3COVLIFGAULIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.901472728 (550h13m29.772555773s) [noise=true , address=45.91.116.5:4299 , piece_id=CN5VEXV4ZGVZTPPO3YB2TMOA5OV4JOCSVDWLY6I2K23PKRAO4TZQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905337859 (550h13m29.776420894s) [noise=true , address=45.91.116.6:4974 , piece_id=GEMMD7VZYWHZLG4B76APXHSB7KGVPG3MSDEF57SJVO5TAPMW4JRA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: <nil> Dec 28 13:42:29.905361523 (550h13m29.776444568s) [seg_pos={0 0}, upload=482419]: Segment reached optimal threshold of 13 pieces. Dec 28 13:42:29.905586202 (550h13m29.776669237s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52224->45.91.116.7:4012: use of closed network connection Dec 28 13:42:29.905593856 (550h13m29.776676891s) [noise=true , address=45.91.116.7:4012 , piece_id=AXOK3PD67ZRNFIUPYQCRHKJHJWJQKXQGERFD74VAUBPMD4PCXQKA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.905731959 (550h13m29.776814994s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 13:42:29.905738521 (550h13m29.776821556s) [noise=true , address=145.79.227.33:4869 , piece_id=4YJ377PHQUZ7ZTKIQ7Q3VSSNDU7K3NRAFVRIPGDR6RK2ZI5Y44YA, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906284732 (550h13m29.777367777s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: read tcp 188.166.225.170:58736->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:42:29.906304299 (550h13m29.777387344s) [noise=true , address=45.91.116.14:4192 , piece_id=XVNBBENVDOPV7E6GV24SWSVNQREGN6EERNVKWWZWSGTT2FGZXFIQ, seg_pos={0 0}, upload=482419]: Long tail cancellation due to optimal threshold Dec 28 13:42:29.906332723 (550h13m29.777415768s) [seg_pos={0 0}, upload=482419]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:42:29.906459084 (550h13m29.777542129s) [upload=482419]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:42:29.906466838 (550h13m29.777549883s) [upload=482419]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:42:30.209359044 (550h13m30.080442089s) [upload=482419]: Done uploading segments... Dec 28 13:42:30.209387108 (550h13m30.080470143s) [upload=482419]: Upload finished. err: <nil> Duration: 30.004382579s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..XX....X...........X....................X............X.....] duration: 21.713634857s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:42:49 UTC) orbiter warning: context deadline exceeded Duration: 21.71365744s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 50 out of 60 is failed (threshold=6) [X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXXX] duration: 5.001079401s trace: https://jaeger.dev.storj.io/trace/327d2765859fe805 FAIL (2025-12-28 13:42:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001100351s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 4.966535693s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:42:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 4.966568345s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXXX] duration: 5.00108579s trace: https://jaeger.dev.storj.io/trace/129ca30ddde0bf8a FAIL (2025-12-28 13:42:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001107483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...XX....X...........X....................X............X....] duration: 16.100693804s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:42:12 UTC) orbiter warning: context deadline exceeded Duration: 16.100720675s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.496251145s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:42:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 8.496272225s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X.XX..XX..X.......XXX......XX......X..X.....X..X.X...X] duration: 30.005042664s trace: https://jaeger.dev.storj.io/trace/63c3258dc449b28b FAIL (2025-12-28 13:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: OG522YL4HNW233TUEEIFRHDUISNBFEO35BY6QXNPX7N7U5R6MKRA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: EHC25BBJYLRHWQHKAOAPCZ4EW5IZLFQXVOKIWA5KJLEUH53UYSBA) context deadline exceeded ======LOGS====== Dec 28 13:41:34.556908970 (550h12m34.427992015s) [upload=482405]: Starting upload Dec 28 13:41:34.556985415 (550h12m34.42806846s) [upload=482405]: Done starting upload Dec 28 13:41:34.557055939 (550h12m34.428138974s) [upload=482405]: Uploading segments... Dec 28 13:41:34.557208110 (550h12m34.428291145s) [upload=482405]: Got next segment. Inline: false Dec 28 13:41:34.557254177 (550h12m34.428337212s) [upload=482405]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:41:34.557264426 (550h12m34.428347461s) [upload=482405]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:41:34.804863226 (550h12m34.675946261s) [seg_pos={0 0}, upload=482405]: Begin upload segment... Dec 28 13:41:34.804913100 (550h12m34.675996135s) [seg_pos={0 0}, upload=482405]: Done begin upload segment. Dec 28 13:41:34.804947225 (550h12m34.67603027s) [noise=true , address=145.79.227.15:4873 , piece_id=NFCONST5GZCSYNQLEOKGAPJP6ADOQATCZBVQK75LXZ6SM3Z2IILA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.804961653 (550h12m34.676044688s) [noise=true , address=77.90.31.13:4626 , piece_id=KHZOEKSDB6A6WIHIIVZ3NIYL3MMG654LAOB5RBPTO7YTRFRXR67Q, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.805072394 (550h12m34.676155439s) [noise=true , address=145.79.227.4:4187 , piece_id=O3BXQSAIEWLNJP2ANT6S22XMWHVZ65AMEXFKIG6CWNI6UQY73DAA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.805400869 (550h12m34.676483914s) [noise=true , address=145.79.227.10:4401 , piece_id=P5ZZQTJSBXPYEXOQZ6IVKLYLF64M3IGEJBBYLCV6NOGVLJHKEPGQ, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.806397677 (550h12m34.677480712s) [noise=true , address=45.91.116.7:4009 , piece_id=EHC25BBJYLRHWQHKAOAPCZ4EW5IZLFQXVOKIWA5KJLEUH53UYSBA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.807065809 (550h12m34.678148844s) [noise=true , address=145.79.227.20:4439 , piece_id=DHAXR7KRABZJCJQNC56G2UF4QHBLJ6AIZR5DX5AWEU4TXTBQOSYQ, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.807392271 (550h12m34.678475306s) [noise=true , address=145.79.227.46:4603 , piece_id=GGNSNZVO2Z4UHI6VW2VQB2HYKSRFAZA7HNNWRQCSETON52SAWWCA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.808598618 (550h12m34.679681653s) [noise=true , address=145.79.227.16:4911 , piece_id=6T6FPBHAA3Q7ADHCCKVFFEFMJ2ZVYDKOARL4MF2UXWF4H6P4ELHQ, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.809683854 (550h12m34.680766889s) [noise=true , address=77.90.31.19:4841 , piece_id=IMFEENHJRWVVHHOWLH75ZI2DRFWL6THLIKOZYBTUGXLTIJPPBJLA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.811326472 (550h12m34.682409517s) [noise=true , address=45.91.116.9:4261 , piece_id=DWEXO45YSVOECPRSZP7OCMIIGKEFHBSNE7KZ3PGZKJ2422FAQ5BA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.812410938 (550h12m34.683493973s) [noise=true , address=145.79.227.47:4267 , piece_id=SFTMGN3YWO6W72G4CEG6XPDU2CL476YEY32KWQOMHX4EDLOTYNWA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.813508557 (550h12m34.684591592s) [noise=true , address=145.79.227.5:4221 , piece_id=LDPX4DRSNPMQFWABL37OZQ5ANRLNX47623VRP6C5YTRGYXYRHCEQ, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.813826012 (550h12m34.684909057s) [noise=true , address=45.91.116.6:4973 , piece_id=UT3ZVRPVX66S6POFKLD5365K42XHG3OTBAFLS3IDVQMXVE7NBSHQ, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.815160604 (550h12m34.686243649s) [noise=true , address=45.91.116.10:4082 , piece_id=OG522YL4HNW233TUEEIFRHDUISNBFEO35BY6QXNPX7N7U5R6MKRA, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.815449784 (550h12m34.686532819s) [noise=true , address=145.79.227.39:4015 , piece_id=JCEQRT2TGJ6W5W5NO2UFV6D6433QCVNUMTPWNOMMNCMRMZ3GRDRQ, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:34.816573463 (550h12m34.687656498s) [noise=true , address=145.79.227.11:4731 , piece_id=GZRE672EQD4ORZ4Q6CWXY2SIPZRI6ONP6UUGEGY6OS42K3WJBF7Q, seg_pos={0 0}, upload=482405]: Uploading piece... Dec 28 13:41:36.529772304 (550h12m36.400855349s) [upload=482405]: Next returned nil segment Dec 28 13:41:36.529825164 (550h12m36.400908199s) [upload=482405]: Waiting for error group managing segments... Dec 28 13:41:36.817288942 (550h12m36.688371977s) [noise=true , address=145.79.227.20:4439 , piece_id=DHAXR7KRABZJCJQNC56G2UF4QHBLJ6AIZR5DX5AWEU4TXTBQOSYQ, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.818202041 (550h12m36.689285076s) [noise=true , address=145.79.227.46:4603 , piece_id=GGNSNZVO2Z4UHI6VW2VQB2HYKSRFAZA7HNNWRQCSETON52SAWWCA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.825843882 (550h12m36.696926927s) [noise=true , address=145.79.227.10:4401 , piece_id=P5ZZQTJSBXPYEXOQZ6IVKLYLF64M3IGEJBBYLCV6NOGVLJHKEPGQ, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.829547043 (550h12m36.700630089s) [noise=true , address=145.79.227.11:4731 , piece_id=GZRE672EQD4ORZ4Q6CWXY2SIPZRI6ONP6UUGEGY6OS42K3WJBF7Q, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.835282425 (550h12m36.70636547s) [noise=true , address=145.79.227.4:4187 , piece_id=O3BXQSAIEWLNJP2ANT6S22XMWHVZ65AMEXFKIG6CWNI6UQY73DAA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.842726370 (550h12m36.713809405s) [noise=true , address=145.79.227.16:4911 , piece_id=6T6FPBHAA3Q7ADHCCKVFFEFMJ2ZVYDKOARL4MF2UXWF4H6P4ELHQ, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.844850424 (550h12m36.715933469s) [noise=true , address=145.79.227.39:4015 , piece_id=JCEQRT2TGJ6W5W5NO2UFV6D6433QCVNUMTPWNOMMNCMRMZ3GRDRQ, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.851109301 (550h12m36.722192346s) [noise=true , address=145.79.227.47:4267 , piece_id=SFTMGN3YWO6W72G4CEG6XPDU2CL476YEY32KWQOMHX4EDLOTYNWA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:36.877544999 (550h12m36.748628044s) [noise=true , address=145.79.227.5:4221 , piece_id=LDPX4DRSNPMQFWABL37OZQ5ANRLNX47623VRP6C5YTRGYXYRHCEQ, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:37.063952420 (550h12m36.935035465s) [noise=true , address=45.91.116.10:4082 , piece_id=OG522YL4HNW233TUEEIFRHDUISNBFEO35BY6QXNPX7N7U5R6MKRA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:37.113603784 (550h12m36.98468683s) [noise=true , address=45.91.116.6:4973 , piece_id=UT3ZVRPVX66S6POFKLD5365K42XHG3OTBAFLS3IDVQMXVE7NBSHQ, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:37.168367613 (550h12m37.039450658s) [noise=true , address=45.91.116.7:4009 , piece_id=EHC25BBJYLRHWQHKAOAPCZ4EW5IZLFQXVOKIWA5KJLEUH53UYSBA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:37.172235869 (550h12m37.043318904s) [noise=true , address=77.90.31.19:4841 , piece_id=IMFEENHJRWVVHHOWLH75ZI2DRFWL6THLIKOZYBTUGXLTIJPPBJLA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: <nil> Dec 28 13:41:37.172262760 (550h12m37.043345795s) [seg_pos={0 0}, upload=482405]: Segment reached optimal threshold of 13 pieces. Dec 28 13:41:37.172502427 (550h12m37.043585482s) [noise=true , address=45.91.116.9:4261 , piece_id=DWEXO45YSVOECPRSZP7OCMIIGKEFHBSNE7KZ3PGZKJ2422FAQ5BA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 13:41:37.172512976 (550h12m37.043596042s) [noise=true , address=45.91.116.9:4261 , piece_id=DWEXO45YSVOECPRSZP7OCMIIGKEFHBSNE7KZ3PGZKJ2422FAQ5BA, seg_pos={0 0}, upload=482405]: Long tail cancellation due to optimal threshold Dec 28 13:41:37.172535189 (550h12m37.043618254s) [noise=true , address=77.90.31.13:4626 , piece_id=KHZOEKSDB6A6WIHIIVZ3NIYL3MMG654LAOB5RBPTO7YTRFRXR67Q, seg_pos={0 0}, upload=482405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 28 13:41:37.172542944 (550h12m37.043625979s) [noise=true , address=77.90.31.13:4626 , piece_id=KHZOEKSDB6A6WIHIIVZ3NIYL3MMG654LAOB5RBPTO7YTRFRXR67Q, seg_pos={0 0}, upload=482405]: Long tail cancellation due to optimal threshold Dec 28 13:41:37.172872050 (550h12m37.043955105s) [noise=true , address=145.79.227.15:4873 , piece_id=NFCONST5GZCSYNQLEOKGAPJP6ADOQATCZBVQK75LXZ6SM3Z2IILA, seg_pos={0 0}, upload=482405]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: expected piece hash; context canceled; EOF Dec 28 13:41:37.172882440 (550h12m37.043965495s) [noise=true , address=145.79.227.15:4873 , piece_id=NFCONST5GZCSYNQLEOKGAPJP6ADOQATCZBVQK75LXZ6SM3Z2IILA, seg_pos={0 0}, upload=482405]: Long tail cancellation due to optimal threshold Dec 28 13:41:37.172925853 (550h12m37.044008888s) [seg_pos={0 0}, upload=482405]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:41:37.173035902 (550h12m37.044118937s) [upload=482405]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:41:37.173043316 (550h12m37.044126351s) [upload=482405]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:41:37.466470139 (550h12m37.337553184s) [upload=482405]: Done uploading segments... Dec 28 13:41:37.466520374 (550h12m37.33760341s) [upload=482405]: Upload finished. err: <nil> Duration: 30.005071868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X........X...................] duration: 43.725499326s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:42:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 43.725521419s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....XX....X...........X....................X............X...] duration: 15.064247117s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:41:53 UTC) orbiter warning: context deadline exceeded Duration: 15.064274598s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [X.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXXX] duration: 5.000868025s trace: https://jaeger.dev.storj.io/trace/e19edbcdc1dca34 FAIL (2025-12-28 13:41:24 UTC) orbiter warning: context deadline exceeded Duration: 5.000890999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......X.XX..XX..X.......XXX......XX......X..X.....X..X.X...] duration: 5.978659359s trace: https://jaeger.dev.storj.io/trace/26158bc7c8d59549 FAIL (2025-12-28 13:41:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ) context deadline exceeded ======LOGS====== .087384583 (550h9m59.958467628s) [upload=482365]: Uploading segments... Dec 28 13:39:00.087526954 (550h9m59.958609999s) [upload=482365]: Got next segment. Inline: false Dec 28 13:39:00.087562081 (550h9m59.958645126s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:39:00.087567090 (550h9m59.958650125s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:39:00.339291928 (550h10m0.210374963s) [seg_pos={0 0}, upload=482365]: Begin upload segment... Dec 28 13:39:00.339341051 (550h10m0.210424086s) [seg_pos={0 0}, upload=482365]: Done begin upload segment. Dec 28 13:39:00.339370307 (550h10m0.210453342s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339413449 (550h10m0.210496494s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339901739 (550h10m0.210984774s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340204696 (550h10m0.211287731s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340468267 (550h10m0.211551302s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340833674 (550h10m0.211916709s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341243013 (550h10m0.212326048s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341630541 (550h10m0.212713576s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.342540123 (550h10m0.213623218s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.343844025 (550h10m0.21492706s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.344200365 (550h10m0.2152834s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.345445635 (550h10m0.21652868s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.346547123 (550h10m0.217630168s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.348124015 (550h10m0.21920705s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.349201707 (550h10m0.220284743s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.350445085 (550h10m0.22152813s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:02.003767658 (550h10m1.874850713s) [upload=482365]: Next returned nil segment Dec 28 13:39:02.003785994 (550h10m1.874869039s) [upload=482365]: Waiting for error group managing segments... Dec 28 13:39:02.387103923 (550h10m2.258186968s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.387465331 (550h10m2.258548376s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392733522 (550h10m2.263816577s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392796292 (550h10m2.263879347s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.393370626 (550h10m2.264453661s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.412180693 (550h10m2.283263738s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.413301146 (550h10m2.284384191s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.415350980 (550h10m2.286434025s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.419081844 (550h10m2.290164889s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.421921681 (550h10m2.293004726s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.647096561 (550h10m2.518179616s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.678500959 (550h10m2.549584004s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749321734 (550h10m2.620404779s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749347002 (550h10m2.620430047s) [seg_pos={0 0}, upload=482365]: Segment reached optimal threshold of 13 pieces. Dec 28 13:39:02.749620673 (550h10m2.620703718s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection Dec 28 13:39:02.749633568 (550h10m2.620716603s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.750306429 (550h10m2.621389475s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection Dec 28 13:39:02.750324634 (550h10m2.621407669s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751063871 (550h10m2.622146916s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: context canceled; context canceled Dec 28 13:39:02.751077377 (550h10m2.622160422s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751108266 (550h10m2.622191311s) [seg_pos={0 0}, upload=482365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:39:02.751207003 (550h10m2.622290049s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:39:02.751213847 (550h10m2.622296892s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:39:03.043703946 (550h10m2.914787001s) [upload=482365]: Done uploading segments... Dec 28 13:39:03.043731558 (550h10m2.914814603s) [upload=482365]: Upload finished. err: <nil> Duration: 5.978683894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X........X..................] duration: 46.098694253s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:41:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 46.098713521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 11.669037847s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:41:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 11.669070318s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........X.XX..XX..X.......XXX......XX......X..X.....X..X.X..] duration: 5.733871645s trace: https://jaeger.dev.storj.io/trace/26158bc7c8d59549 FAIL (2025-12-28 13:40:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ) context deadline exceeded ======LOGS====== .087384583 (550h9m59.958467628s) [upload=482365]: Uploading segments... Dec 28 13:39:00.087526954 (550h9m59.958609999s) [upload=482365]: Got next segment. Inline: false Dec 28 13:39:00.087562081 (550h9m59.958645126s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:39:00.087567090 (550h9m59.958650125s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:39:00.339291928 (550h10m0.210374963s) [seg_pos={0 0}, upload=482365]: Begin upload segment... Dec 28 13:39:00.339341051 (550h10m0.210424086s) [seg_pos={0 0}, upload=482365]: Done begin upload segment. Dec 28 13:39:00.339370307 (550h10m0.210453342s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339413449 (550h10m0.210496494s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339901739 (550h10m0.210984774s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340204696 (550h10m0.211287731s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340468267 (550h10m0.211551302s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340833674 (550h10m0.211916709s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341243013 (550h10m0.212326048s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341630541 (550h10m0.212713576s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.342540123 (550h10m0.213623218s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.343844025 (550h10m0.21492706s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.344200365 (550h10m0.2152834s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.345445635 (550h10m0.21652868s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.346547123 (550h10m0.217630168s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.348124015 (550h10m0.21920705s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.349201707 (550h10m0.220284743s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.350445085 (550h10m0.22152813s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:02.003767658 (550h10m1.874850713s) [upload=482365]: Next returned nil segment Dec 28 13:39:02.003785994 (550h10m1.874869039s) [upload=482365]: Waiting for error group managing segments... Dec 28 13:39:02.387103923 (550h10m2.258186968s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.387465331 (550h10m2.258548376s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392733522 (550h10m2.263816577s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392796292 (550h10m2.263879347s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.393370626 (550h10m2.264453661s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.412180693 (550h10m2.283263738s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.413301146 (550h10m2.284384191s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.415350980 (550h10m2.286434025s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.419081844 (550h10m2.290164889s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.421921681 (550h10m2.293004726s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.647096561 (550h10m2.518179616s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.678500959 (550h10m2.549584004s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749321734 (550h10m2.620404779s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749347002 (550h10m2.620430047s) [seg_pos={0 0}, upload=482365]: Segment reached optimal threshold of 13 pieces. Dec 28 13:39:02.749620673 (550h10m2.620703718s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection Dec 28 13:39:02.749633568 (550h10m2.620716603s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.750306429 (550h10m2.621389475s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection Dec 28 13:39:02.750324634 (550h10m2.621407669s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751063871 (550h10m2.622146916s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: context canceled; context canceled Dec 28 13:39:02.751077377 (550h10m2.622160422s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751108266 (550h10m2.622191311s) [seg_pos={0 0}, upload=482365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:39:02.751207003 (550h10m2.622290049s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:39:02.751213847 (550h10m2.622296892s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:39:03.043703946 (550h10m2.914787001s) [upload=482365]: Done uploading segments... Dec 28 13:39:03.043731558 (550h10m2.914814603s) [upload=482365]: Upload finished. err: <nil> Duration: 5.733899268s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....XX....X...........X....................X............X..] duration: 15.158931072s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:40:40 UTC) orbiter warning: context deadline exceeded Duration: 15.158962081s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X........X.................] duration: 44.214245569s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:40:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 44.214276657s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 49 out of 60 is failed (threshold=6) [XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXXX] duration: 5.00057608s trace: https://jaeger.dev.storj.io/trace/52a9f42bead7de90 FAIL (2025-12-28 13:40:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000608692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 15.47431291s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:40:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 15.47434441s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........X.XX..XX..X.......XXX......XX......X..X.....X..X.X.] duration: 6.192292393s trace: https://jaeger.dev.storj.io/trace/26158bc7c8d59549 FAIL (2025-12-28 13:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ) context deadline exceeded ======LOGS====== .087384583 (550h9m59.958467628s) [upload=482365]: Uploading segments... Dec 28 13:39:00.087526954 (550h9m59.958609999s) [upload=482365]: Got next segment. Inline: false Dec 28 13:39:00.087562081 (550h9m59.958645126s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:39:00.087567090 (550h9m59.958650125s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:39:00.339291928 (550h10m0.210374963s) [seg_pos={0 0}, upload=482365]: Begin upload segment... Dec 28 13:39:00.339341051 (550h10m0.210424086s) [seg_pos={0 0}, upload=482365]: Done begin upload segment. Dec 28 13:39:00.339370307 (550h10m0.210453342s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339413449 (550h10m0.210496494s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339901739 (550h10m0.210984774s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340204696 (550h10m0.211287731s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340468267 (550h10m0.211551302s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340833674 (550h10m0.211916709s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341243013 (550h10m0.212326048s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341630541 (550h10m0.212713576s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.342540123 (550h10m0.213623218s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.343844025 (550h10m0.21492706s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.344200365 (550h10m0.2152834s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.345445635 (550h10m0.21652868s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.346547123 (550h10m0.217630168s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.348124015 (550h10m0.21920705s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.349201707 (550h10m0.220284743s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.350445085 (550h10m0.22152813s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:02.003767658 (550h10m1.874850713s) [upload=482365]: Next returned nil segment Dec 28 13:39:02.003785994 (550h10m1.874869039s) [upload=482365]: Waiting for error group managing segments... Dec 28 13:39:02.387103923 (550h10m2.258186968s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.387465331 (550h10m2.258548376s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392733522 (550h10m2.263816577s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392796292 (550h10m2.263879347s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.393370626 (550h10m2.264453661s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.412180693 (550h10m2.283263738s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.413301146 (550h10m2.284384191s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.415350980 (550h10m2.286434025s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.419081844 (550h10m2.290164889s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.421921681 (550h10m2.293004726s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.647096561 (550h10m2.518179616s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.678500959 (550h10m2.549584004s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749321734 (550h10m2.620404779s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749347002 (550h10m2.620430047s) [seg_pos={0 0}, upload=482365]: Segment reached optimal threshold of 13 pieces. Dec 28 13:39:02.749620673 (550h10m2.620703718s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection Dec 28 13:39:02.749633568 (550h10m2.620716603s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.750306429 (550h10m2.621389475s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection Dec 28 13:39:02.750324634 (550h10m2.621407669s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751063871 (550h10m2.622146916s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: context canceled; context canceled Dec 28 13:39:02.751077377 (550h10m2.622160422s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751108266 (550h10m2.622191311s) [seg_pos={0 0}, upload=482365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:39:02.751207003 (550h10m2.622290049s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:39:02.751213847 (550h10m2.622296892s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:39:03.043703946 (550h10m2.914787001s) [upload=482365]: Done uploading segments... Dec 28 13:39:03.043731558 (550h10m2.914814603s) [upload=482365]: Upload finished. err: <nil> Duration: 6.192320678s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXXX] duration: 5.00154199s trace: https://jaeger.dev.storj.io/trace/72a5bebb8c4e0554 FAIL (2025-12-28 13:40:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001568501s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.....XX....X...........X....................X............X.] duration: 23.508414011s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:39:59 UTC) orbiter warning: context deadline exceeded Duration: 23.508432367s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXXX] duration: 5.001560005s trace: https://jaeger.dev.storj.io/trace/42bbeeabc4970fc1 FAIL (2025-12-28 13:39:31 UTC) orbiter warning: context deadline exceeded Duration: 5.001585722s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..........X.XX..XX..X.......XXX......XX......X..X.....X..X.X] duration: 30.003031851s trace: https://jaeger.dev.storj.io/trace/26158bc7c8d59549 FAIL (2025-12-28 13:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ) context deadline exceeded ======LOGS====== .087384583 (550h9m59.958467628s) [upload=482365]: Uploading segments... Dec 28 13:39:00.087526954 (550h9m59.958609999s) [upload=482365]: Got next segment. Inline: false Dec 28 13:39:00.087562081 (550h9m59.958645126s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:39:00.087567090 (550h9m59.958650125s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:39:00.339291928 (550h10m0.210374963s) [seg_pos={0 0}, upload=482365]: Begin upload segment... Dec 28 13:39:00.339341051 (550h10m0.210424086s) [seg_pos={0 0}, upload=482365]: Done begin upload segment. Dec 28 13:39:00.339370307 (550h10m0.210453342s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339413449 (550h10m0.210496494s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.339901739 (550h10m0.210984774s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340204696 (550h10m0.211287731s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340468267 (550h10m0.211551302s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.340833674 (550h10m0.211916709s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341243013 (550h10m0.212326048s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.341630541 (550h10m0.212713576s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.342540123 (550h10m0.213623218s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.343844025 (550h10m0.21492706s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.344200365 (550h10m0.2152834s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.345445635 (550h10m0.21652868s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.346547123 (550h10m0.217630168s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.348124015 (550h10m0.21920705s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.349201707 (550h10m0.220284743s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:00.350445085 (550h10m0.22152813s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Uploading piece... Dec 28 13:39:02.003767658 (550h10m1.874850713s) [upload=482365]: Next returned nil segment Dec 28 13:39:02.003785994 (550h10m1.874869039s) [upload=482365]: Waiting for error group managing segments... Dec 28 13:39:02.387103923 (550h10m2.258186968s) [noise=true , address=145.79.227.52:4446 , piece_id=7BNTIMHD7FTCZ7MS5KIFY4LBDKJ7MWUMRI7KWPWQTSNFC6DDENJA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.387465331 (550h10m2.258548376s) [noise=true , address=145.79.227.7:4294 , piece_id=54A62U46NQ72JGGN3MFEA2GRM5IQKBCCBLRHQST5M5KXWJXESGUQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392733522 (550h10m2.263816577s) [noise=true , address=145.79.227.51:4410 , piece_id=YRAXLYS3EU4P3RCHQ5ZZUF37JPS3UESAAH26L5KTKMDUSRXKZ3WA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.392796292 (550h10m2.263879347s) [noise=true , address=145.79.227.47:4265 , piece_id=2ZGTZ62VYJNGAFUVGOZIN7I5ZPXCRXPZMX5W6I6SHEQBZN63QKPQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.393370626 (550h10m2.264453661s) [noise=true , address=145.79.227.19:4078 , piece_id=PNLF7GQ3YOSXFO2CQA4CQGTY276GAN2BE3H3MB3E64PGT7LFXFCA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.412180693 (550h10m2.283263738s) [noise=true , address=145.79.227.18:4007 , piece_id=BVHTJA4Z4O4XW7NU5GEZLZPO5UGKE5NSBVNFUEPBAMM47GCDX4XQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.413301146 (550h10m2.284384191s) [noise=true , address=145.79.227.2:4114 , piece_id=JEFWTGIEEV53SUPNNEX773X56SLRUT4GYFQ7HJFQFDRF7TLEWFFQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.415350980 (550h10m2.286434025s) [noise=true , address=145.79.227.45:4230 , piece_id=N3G5GBW73DMC2D33MLAMJ6QGR7YDMTEGR64WH74IR5VSRVAFFPTA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.419081844 (550h10m2.290164889s) [noise=true , address=145.79.227.13:4803 , piece_id=EORCFMDTYAPA2A67V2T2BROS5VJ5A43MYO4X4MDXCVZK7SABKWQQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.421921681 (550h10m2.293004726s) [noise=true , address=145.79.227.16:4911 , piece_id=RSWJP7HJRYL54IBIQ2EGKAOAWCN7VCWWAH2XQFVG7PKJNYFSC7MA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.647096561 (550h10m2.518179616s) [noise=true , address=77.90.31.19:4842 , piece_id=W5SEYB7HWHOWNBSACLXZUKVHAUG2HAQSQ3HI7GXDZDM7XVN24OTQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.678500959 (550h10m2.549584004s) [noise=true , address=45.91.116.13:4156 , piece_id=XFY3UM5OTWK7K7GCFJECVM4IELDOKQA5WSE2WXORLBYKFVGR27ZA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749321734 (550h10m2.620404779s) [noise=true , address=45.91.116.12:4226 , piece_id=T243SH5HMX2NNW5F234PWLO2BIUAIFFR4PS3AD644CYVDVDY2VHA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: <nil> Dec 28 13:39:02.749347002 (550h10m2.620430047s) [seg_pos={0 0}, upload=482365]: Segment reached optimal threshold of 13 pieces. Dec 28 13:39:02.749620673 (550h10m2.620703718s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35428->45.91.116.8:4047: use of closed network connection Dec 28 13:39:02.749633568 (550h10m2.620716603s) [noise=true , address=45.91.116.8:4047 , piece_id=GNQJOY6AMCBMSJAZVECUXIH3ZKYZ7HYVNQRWMBQ5JRQBMD3NQ4FA, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.750306429 (550h10m2.621389475s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50154->77.90.31.12:4589: use of closed network connection Dec 28 13:39:02.750324634 (550h10m2.621407669s) [noise=true , address=77.90.31.12:4589 , piece_id=HABNTJ72PM3XKPMKKFU2OZ7XDVOQ4PUAXDQ7F53U2S6PGON6GL4Q, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751063871 (550h10m2.622146916s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: context canceled; context canceled Dec 28 13:39:02.751077377 (550h10m2.622160422s) [noise=true , address=45.91.116.9:4263 , piece_id=IDP32E6ROULKNJJ2MXKC62KGITDO72BUSBYDKDSMD5Y4HWGQK4PQ, seg_pos={0 0}, upload=482365]: Long tail cancellation due to optimal threshold Dec 28 13:39:02.751108266 (550h10m2.622191311s) [seg_pos={0 0}, upload=482365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:39:02.751207003 (550h10m2.622290049s) [upload=482365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:39:02.751213847 (550h10m2.622296892s) [upload=482365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:39:03.043703946 (550h10m2.914787001s) [upload=482365]: Done uploading segments... Dec 28 13:39:03.043731558 (550h10m2.914814603s) [upload=482365]: Upload finished. err: <nil> Duration: 30.003056489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X........X................] duration: 49.815938062s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 49.81596335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 7.215213942s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:39:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 7.215235032s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.....XX....X...........X....................X............X] duration: 30.001806431s trace: https://jaeger.dev.storj.io/trace/303958fb6bfcc95e FAIL (2025-12-28 13:39:14 UTC) orbiter warning: context deadline exceeded Duration: 30.001828395s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..........X.XX..XX..X.......XXX......XX......X..X.....X..X.] duration: 6.717108028s trace: https://jaeger.dev.storj.io/trace/6c1591a05ad2eac6 FAIL (2025-12-28 13:38:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:38:08.464226011 (550h9m8.335309046s) [upload=482354]: Starting upload Dec 28 13:38:08.464290404 (550h9m8.335373449s) [upload=482354]: Done starting upload Dec 28 13:38:08.464378000 (550h9m8.335461045s) [upload=482354]: Uploading segments... Dec 28 13:38:08.464567011 (550h9m8.335650056s) [upload=482354]: Got next segment. Inline: false Dec 28 13:38:08.464622396 (550h9m8.335705441s) [upload=482354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:38:08.464627546 (550h9m8.335710581s) [upload=482354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:38:08.711212654 (550h9m8.582295689s) [seg_pos={0 0}, upload=482354]: Begin upload segment... Dec 28 13:38:08.711289841 (550h9m8.582372876s) [seg_pos={0 0}, upload=482354]: Done begin upload segment. Dec 28 13:38:08.711362389 (550h9m8.582445434s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.711400963 (550h9m8.582484008s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.712394153 (550h9m8.583477198s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.712489735 (550h9m8.583572781s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.713144693 (550h9m8.584227738s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.713619085 (550h9m8.584702131s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.713652098 (550h9m8.584735133s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.714338365 (550h9m8.585421411s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.714603731 (550h9m8.585686776s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.714665359 (550h9m8.585748394s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.715357636 (550h9m8.586440681s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.715714086 (550h9m8.586797121s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.717021004 (550h9m8.588104049s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.719023978 (550h9m8.590107013s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.723965698 (550h9m8.595048743s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.732674431 (550h9m8.603757496s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:10.431589151 (550h9m10.302672196s) [upload=482354]: Next returned nil segment Dec 28 13:38:10.431644125 (550h9m10.302727161s) [upload=482354]: Waiting for error group managing segments... Dec 28 13:38:10.727468924 (550h9m10.598551969s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.728722090 (550h9m10.599805135s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.737014090 (550h9m10.608097135s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.741090482 (550h9m10.612173527s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.745833063 (550h9m10.616916108s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.749513942 (550h9m10.620596987s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.749569217 (550h9m10.620652252s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.754150741 (550h9m10.625233776s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.765496985 (550h9m10.636580031s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.770863013 (550h9m10.641946058s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.771430154 (550h9m10.642513199s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.951904947 (550h9m10.822987993s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:11.007731691 (550h9m10.878814736s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:11.007758952 (550h9m10.878841997s) [seg_pos={0 0}, upload=482354]: Segment reached optimal threshold of 13 pieces. Dec 28 13:38:11.008042753 (550h9m10.879125788s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 13:38:11.008050788 (550h9m10.879133833s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold Dec 28 13:38:11.008219600 (550h9m10.879302645s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 13:38:11.008226793 (550h9m10.879309828s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold Dec 28 13:38:11.008453564 (550h9m10.879536609s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43300->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:38:11.008461540 (550h9m10.879544585s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold Dec 28 13:38:11.008487138 (550h9m10.879570183s) [seg_pos={0 0}, upload=482354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:38:11.008569305 (550h9m10.87965235s) [upload=482354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:38:11.008574235 (550h9m10.87965728s) [upload=482354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:38:11.296594462 (550h9m11.167677507s) [upload=482354]: Done uploading segments... Dec 28 13:38:11.296702798 (550h9m11.167785833s) [upload=482354]: Upload finished. err: <nil> Duration: 6.717141482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..........X.XX..XX..X.......XXX......XX......X..X.....X..X] duration: 30.002879161s trace: https://jaeger.dev.storj.io/trace/6c1591a05ad2eac6 FAIL (2025-12-28 13:38:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 13:38:08.464226011 (550h9m8.335309046s) [upload=482354]: Starting upload Dec 28 13:38:08.464290404 (550h9m8.335373449s) [upload=482354]: Done starting upload Dec 28 13:38:08.464378000 (550h9m8.335461045s) [upload=482354]: Uploading segments... Dec 28 13:38:08.464567011 (550h9m8.335650056s) [upload=482354]: Got next segment. Inline: false Dec 28 13:38:08.464622396 (550h9m8.335705441s) [upload=482354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:38:08.464627546 (550h9m8.335710581s) [upload=482354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:38:08.711212654 (550h9m8.582295689s) [seg_pos={0 0}, upload=482354]: Begin upload segment... Dec 28 13:38:08.711289841 (550h9m8.582372876s) [seg_pos={0 0}, upload=482354]: Done begin upload segment. Dec 28 13:38:08.711362389 (550h9m8.582445434s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.711400963 (550h9m8.582484008s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.712394153 (550h9m8.583477198s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.712489735 (550h9m8.583572781s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.713144693 (550h9m8.584227738s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.713619085 (550h9m8.584702131s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.713652098 (550h9m8.584735133s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.714338365 (550h9m8.585421411s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.714603731 (550h9m8.585686776s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.714665359 (550h9m8.585748394s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.715357636 (550h9m8.586440681s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.715714086 (550h9m8.586797121s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.717021004 (550h9m8.588104049s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.719023978 (550h9m8.590107013s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.723965698 (550h9m8.595048743s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:08.732674431 (550h9m8.603757496s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Uploading piece... Dec 28 13:38:10.431589151 (550h9m10.302672196s) [upload=482354]: Next returned nil segment Dec 28 13:38:10.431644125 (550h9m10.302727161s) [upload=482354]: Waiting for error group managing segments... Dec 28 13:38:10.727468924 (550h9m10.598551969s) [noise=true , address=145.79.227.20:4439 , piece_id=ZCVMKKCNYPZIYZ7PXN4RPLYH7HEYYOBDIHGGRZOJOFLVLUMU3YOA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.728722090 (550h9m10.599805135s) [noise=true , address=145.79.227.2:4113 , piece_id=DBRZEZ5GFAN4JYS7UDJHGPBQTQVH4JPP72CKBI45V45OSOGWUJSA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.737014090 (550h9m10.608097135s) [noise=true , address=145.79.227.45:4230 , piece_id=FUNZ44LMQCBRQDFH6PGUUJNAS4I42FQVNUAYGD57AUW7HRNXRFEA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.741090482 (550h9m10.612173527s) [noise=true , address=145.79.227.17:4971 , piece_id=DI6WJ3JG2BF4XAKNIOTDNQH6R2FOD3YNFUGLKECIQFSUOWJLR37Q, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.745833063 (550h9m10.616916108s) [noise=true , address=145.79.227.11:4732 , piece_id=4UFKTE6IXATZGK27J7YYHP536AC3XJZR4LXSRUXKNVLGXQROG2KQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.749513942 (550h9m10.620596987s) [noise=true , address=145.79.227.14:4837 , piece_id=SIARUE6GC2ID72NE2C2M54SRKZBN6C7GIWAOWNJVUM3NM3HJXGXA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.749569217 (550h9m10.620652252s) [noise=true , address=145.79.227.13:4802 , piece_id=C275GJ5EWURGVA7RB5RXIJVTI6ZX3MR6JHY2PJ55OVPZATNVX6NA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.754150741 (550h9m10.625233776s) [noise=true , address=145.79.227.16:4909 , piece_id=7KQVCOKBXNI5GCMIODLK5Q3L2M5BQQDMNE64NELPB5SZD3C4INFQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.765496985 (550h9m10.636580031s) [noise=true , address=145.79.227.5:4223 , piece_id=UQNK7VCNMTIUCT5AULIRQKHRAM6BPBEFTSPCTU7S5HAXK534Z3IA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.770863013 (550h9m10.641946058s) [noise=true , address=145.79.227.47:4267 , piece_id=XT7U4CCX7QFCDTAHYQRPYN3BMCK4ZOVDJD74Y3MM54VN3MYXCQ7A, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.771430154 (550h9m10.642513199s) [noise=true , address=145.79.227.32:4835 , piece_id=MJAJMF2WNHWXD2JBNSXTQCSU6XTBFI42G2KORISJWQVLXOLGBFYA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:10.951904947 (550h9m10.822987993s) [noise=true , address=45.91.116.10:4081 , piece_id=5YJRUJCBXZLZFWFFMUUAW4AH2RUAOJCHC2LNDRPB5TFROKDSVXDQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:11.007731691 (550h9m10.878814736s) [noise=true , address=45.91.116.13:4154 , piece_id=B74XHHHIZIMBMRITY3HFFE6YWBPCGJKPZ6PIAIPTUCG76GWB6SSQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: <nil> Dec 28 13:38:11.007758952 (550h9m10.878841997s) [seg_pos={0 0}, upload=482354]: Segment reached optimal threshold of 13 pieces. Dec 28 13:38:11.008042753 (550h9m10.879125788s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 13:38:11.008050788 (550h9m10.879133833s) [noise=true , address=77.90.31.10:4517 , piece_id=W4SENDFSD2QPGNLAEY53UWNJGDTZOKOPSFFINA4YH63T3ONJC3FA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold Dec 28 13:38:11.008219600 (550h9m10.879302645s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 13:38:11.008226793 (550h9m10.879309828s) [noise=true , address=145.79.227.33:4869 , piece_id=XX4EYXDPMFFRRG7AW6NBZ5MGZJC47DOFEXNQORDEN74BEUHF33DA, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold Dec 28 13:38:11.008453564 (550h9m10.879536609s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43300->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:38:11.008461540 (550h9m10.879544585s) [noise=true , address=77.90.31.16:4736 , piece_id=2XZZNFDZCOX7LVVBH4KMAWC7F3MDA3E2D5DOH72WX2OVZ24GNCGQ, seg_pos={0 0}, upload=482354]: Long tail cancellation due to optimal threshold Dec 28 13:38:11.008487138 (550h9m10.879570183s) [seg_pos={0 0}, upload=482354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:38:11.008569305 (550h9m10.87965235s) [upload=482354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:38:11.008574235 (550h9m10.87965728s) [upload=482354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:38:11.296594462 (550h9m11.167677507s) [upload=482354]: Done uploading segments... Dec 28 13:38:11.296702798 (550h9m11.167785833s) [upload=482354]: Upload finished. err: <nil> Duration: 30.002903997s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....XX....X...........X....................X............] duration: 15.641888775s trace: https://jaeger.dev.storj.io/trace/745a9066fe03a6ff FAIL (2025-12-28 13:38:26 UTC) orbiter warning: context deadline exceeded Duration: 15.642009335s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XXX] duration: 5.001581004s trace: https://jaeger.dev.storj.io/trace/6d2059f6d946a68e FAIL (2025-12-28 13:38:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001605871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.756527322s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:38:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 7.756544755s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....XX....X...........X....................X...........] duration: 21.563481733s trace: https://jaeger.dev.storj.io/trace/745a9066fe03a6ff FAIL (2025-12-28 13:38:09 UTC) orbiter warning: context deadline exceeded Duration: 21.56358106s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004703069s trace: https://jaeger.dev.storj.io/trace/5fe9d6dbb79fbe44 FAIL (2025-12-28 13:38:04 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 67.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.967737953 (550h8m4.838820988s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968193240 (550h8m4.839276285s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.968780638 (550h8m4.839863683s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.969308924 (550h8m4.840391959s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970088349 (550h8m4.841171404s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970325381 (550h8m4.841408426s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.970783813 (550h8m4.841866858s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971205406 (550h8m4.842288441s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.971606299 (550h8m4.842689344s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:04.972162538 (550h8m4.843245583s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Uploading piece... Dec 28 13:37:05.795167814 (550h8m5.666250849s) [upload=482337]: Next returned nil segment Dec 28 13:37:05.795184566 (550h8m5.666267611s) [upload=482337]: Waiting for error group managing segments... Dec 28 13:37:06.178573801 (550h8m6.049656836s) [noise=true , address=185.167.204.206:4049 , piece_id=6RG2DZV6FPTIRE63ZNNST42TSWEBQ456YXIPPVYHIXV55GWFVJ4Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183060835 (550h8m6.05414391s) [noise=true , address=185.167.204.206:4072 , piece_id=WXORBDMA3KW7WZ67A4FJT6URJRFADUJ7X2PAHIGNG47T3TGMXVTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183411202 (550h8m6.054494237s) [noise=true , address=185.167.204.206:4064 , piece_id=OG3G74OU23CEMKA54WQE3DOJ6AKAXBLZ23QWOQC4UEGQQN4JSMXQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183791146 (550h8m6.054874181s) [noise=true , address=185.167.204.205:4029 , piece_id=6JQB6ZSCQZKWH3JWCQ4DY6BBAX73KC4Y53PRSMEMLWURZ5QWWQQQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.183973743 (550h8m6.055056778s) [noise=true , address=185.167.204.206:4053 , piece_id=YDH77S6JLKJ6IYS6DRUCELP5EJHIC2O44QBO7MIOEUYBML4GIBHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184465369 (550h8m6.055548414s) [noise=true , address=185.167.204.205:4008 , piece_id=RTUZPEXBB6JNSMC5YOFQQGDM57YD233W4756ZBQVMFNOVX6YKB4A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184501989 (550h8m6.055585024s) [noise=true , address=185.167.204.205:4030 , piece_id=R5DLI4ZTLOVGO56GPLJL5ZNHYSXOSYNJQHFIO4MVVNHWWBMEOV2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.184856224 (550h8m6.055939259s) [noise=true , address=185.167.204.205:4009 , piece_id=V6RA7Y7FKY4DCPTRRF3BYNBALWDY5VTPXMDUXFE3353KUXLSQJEA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185063518 (550h8m6.056146563s) [noise=true , address=185.167.204.205:4016 , piece_id=JB7SC3J2VR4U33HNQS2A2CA75THT3XLQCOIFHAJJEK7MQGY2SYAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.185288617 (550h8m6.056371652s) [noise=true , address=185.167.204.205:4027 , piece_id=PAPNMRJ7E47VOMOXPBBDWKZJHLRMXTWVR5VW3J7KNQSSMHOHWOAQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.186241401 (550h8m6.057324436s) [noise=true , address=185.167.204.205:4024 , piece_id=FM5IMWRSJJ4RY72IX2OOYC54BFKL4FWE3UOAN3GJ26MLH345COVA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193113658 (550h8m6.064196703s) [noise=true , address=185.167.204.205:4005 , piece_id=R4DFQJZBPJ5WJWHPODV6ILHUDIRBUSAFOMVNCLXB4UNMI6EXTKHA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.193791408 (550h8m6.064874453s) [noise=true , address=185.167.204.206:4068 , piece_id=DAI37M67FWCSVQTQOZ6RSSOJDGUVZO6SNGAHJJ44G3PGS4OJZEMA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194292262 (550h8m6.065375307s) [noise=true , address=185.167.204.206:4046 , piece_id=VWLVHMBMJ63DQEWYBIKH3Y3NTVWFNW5OXZKMYXXBHVI3VWYXLLPQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.194302652 (550h8m6.065385687s) [noise=true , address=185.167.204.206:4067 , piece_id=VSKTG5RH32N7OEVWJGQVB6WJ72TEVF6DDK25W7NOH4RAXUP52LTQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.195359193 (550h8m6.066442228s) [noise=true , address=185.167.204.205:4012 , piece_id=LV6RUA5BWXG4DMSUVZDBGUAF5IBWUGMLJYUKNIXHRVDEBIOW4Q2Q, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.198423448 (550h8m6.069506493s) [noise=true , address=185.167.204.206:4070 , piece_id=5A7YYQYREIZRZDR7TRSE7XFW765JVAV7FX6LKWUI2OXAA74XN4CA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.888968408 (550h8m6.760051453s) [noise=true , address=81.171.14.70:4044 , piece_id=DGN74IG6IWAHLEMO6NLL4TJBXZ6ONTN6QUB6HXY6K7SSO3LYOYIQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.892350789 (550h8m6.763433834s) [noise=true , address=81.171.14.70:4039 , piece_id=EDHPQWLQZTP7MKPGEVMP7JXNTW7LFWSXCTRTSXPCBIQPROX76WRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:06.904477160 (550h8m6.775560195s) [noise=true , address=81.171.14.70:4040 , piece_id=DYF4GSZBAKQ3H63LTVCMEFWMABY3FXHOJCNVIFYRKLF2J6B52K5A, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.128088281 (550h8m6.999171326s) [noise=true , address=81.171.14.70:4036 , piece_id=CICNNV7JE6TBNC4RAPXKCEUSJAADOUVGSVIVQISM3ZB4COAZQDDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.430242142 (550h8m7.301325177s) [noise=true , address=89.149.201.147:4009 , piece_id=EVW2NQFYU57PC2IIVXLYIEBMYXJNIC2CLCHW6PS6NYIPCFRUVHDA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.505304332 (550h8m7.376387367s) [noise=true , address=89.149.201.147:4019 , piece_id=MK7ORCLHO3DOWIOVSOWIQMPJ447PHRQGXP3UKITIRTYJSU5MPCCQ, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.575461543 (550h8m7.446544609s) [noise=true , address=81.171.14.70:4048 , piece_id=TLT47ORGUKGF3MIIEAX7D22VR2FO6W6NL3VFJNHKWD4L3LIMFDLA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.599667265 (550h8m7.4707503s) [noise=true , address=89.149.201.147:4023 , piece_id=5IINX2GVRFLRX2QH24OO2DDANOKOTXTDR6AGVJBGSJ7IAZQ3CASA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928390728 (550h8m7.799473773s) [noise=true , address=89.149.201.147:4002 , piece_id=G2PABNTDWCK4MKOZF6PONBMRNIXI45JTHJMAZFPEKPEJ3PQ7MTBA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: <nil> Dec 28 13:37:07.928417148 (550h8m7.799500193s) [seg_pos={0 0}, upload=482337]: Segment reached optimal threshold of 26 pieces. Dec 28 13:37:07.928656303 (550h8m7.799739348s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZoRwa8wLw66Pjumy1ybp6wrZ79RrSU55uYZpwaY3xi95iSYde): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928664990 (550h8m7.799748035s) [noise=true , address=81.171.14.70:4030 , piece_id=74YDIKUST7X3JBXVQHVLFHIKH5A4DYH5TNL3I4XUA5A6AHQO2DRA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928723351 (550h8m7.799806386s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr): protocol: expected piece hash; context canceled; EOF Dec 28 13:37:07.928733922 (550h8m7.799816957s) [noise=true , address=89.149.201.147:4014 , piece_id=RTC7WV2NW35ODPE6RDQKBWJWSN7FKXFZO3P4P55HHWULIWUKENXA, seg_pos={0 0}, upload=482337]: Long tail cancellation due to optimal threshold Dec 28 13:37:07.928754921 (550h8m7.799837966s) [seg_pos={0 0}, upload=482337]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 28 13:37:07.928870170 (550h8m7.799953215s) [upload=482337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:37:07.928878627 (550h8m7.799961672s) [upload=482337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:37:08.166321536 (550h8m8.037404571s) [upload=482337]: Done uploading segments... Dec 28 13:37:08.166341444 (550h8m8.037424479s) [upload=482337]: Upload finished. err: <nil> Duration: 1m0.004732175s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X........X...............] duration: 32.21776502s trace: https://jaeger.dev.storj.io/trace/1dd6ee7a7b5d1268 FAIL (2025-12-28 13:38:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== D75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112121923 (549h54m49.983204968s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112173502 (549h54m49.983256537s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112558134 (549h54m49.983641179s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.112740712 (549h54m49.983823757s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113109835 (549h54m49.98419287s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113353599 (549h54m49.984436634s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113516609 (549h54m49.984599644s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.113876825 (549h54m49.98495986s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114036739 (549h54m49.985119784s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114461970 (549h54m49.985545015s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114546290 (549h54m49.985629335s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.114995155 (549h54m49.9860782s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115068053 (549h54m49.986151098s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115445122 (549h54m49.986528167s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.115602230 (549h54m49.986685275s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:50.116188487 (549h54m49.987271522s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Uploading piece... Dec 28 13:23:51.389652481 (549h54m51.260735517s) [upload=482157]: Next returned nil segment Dec 28 13:23:51.389665275 (549h54m51.260748311s) [upload=482157]: Waiting for error group managing segments... Dec 28 13:23:51.726210533 (549h54m51.597293578s) [noise=true , address=64.49.13.140:4506 , piece_id=72UR6RLNSBOEFEGWSME5OVIY5MEMLZB6I5HARIG3FJFYDMROP4XQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:51.737733374 (549h54m51.608816419s) [noise=true , address=64.49.13.170:4718 , piece_id=I7XJKYMBWN73H76NPK4CSWHK3DKDLUFB34C5VAC2TFTRKAA6UCCQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.256713908 (549h54m52.127796953s) [noise=true , address=77.90.31.17:4790 , piece_id=P232RNOW66DMBVTATDCERNLLROO7GA6HPMQDSOGDIIY5WOCECU3A, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.290919481 (549h54m52.162002526s) [noise=true , address=77.90.31.11:4563 , piece_id=T5TKT3TUOLMSVJZRISFVDIZAQUX2CWHAXUTK2LJ2LYBKDDINVTAQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.937399575 (549h54m52.80848261s) [noise=true , address=145.79.227.34:4927 , piece_id=RTZP5ADL65FU74PWEKC53CBKSUQCECP5U2ZKN5DHMZE5D2HYBBKQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.943048201 (549h54m52.814131236s) [noise=true , address=145.79.227.32:4863 , piece_id=VSJWFNKF7TBD75Z2DTVENLJQEUJWLCPSSMFZW257JVVQ65FRY6UQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.965894485 (549h54m52.83697752s) [noise=true , address=145.79.227.9:4389 , piece_id=SEMTWSQL5KKZAVFLZHT4LO2E65DYONJZBMOZFHH4TZKAWGOZDPIA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:52.978838883 (549h54m52.849921918s) [noise=true , address=77.90.31.15:4731 , piece_id=3LMGAQ3SF6GYM25GT6YNRSMENRL344UJ4HO6XHBEAGNFHRAFBM2Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.523239682 (549h54m53.394322727s) [noise=true , address=77.90.31.16:4755 , piece_id=GNI7WBFUH446NFMLTQUJY74H2GCUYDETQO6SAUFKGOLQDCQQUIFQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.706106399 (549h54m53.577189434s) [noise=true , address=77.90.31.21:4900 , piece_id=YDEHSUN2HVTYKV7ZT2S4SHIOONY3EUU7L5U5R356Z43TMT43CIEA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.860305269 (549h54m53.731388304s) [noise=true , address=145.79.227.15:4884 , piece_id=R7VP4A5GYXPPSPJRMQAJLKC4WPMVUBHBYXIDOIIZPE5QODPCFKXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:53.904758185 (549h54m53.77584122s) [noise=true , address=77.90.31.19:4875 , piece_id=JJISZXM4BN3FY7VC6ZEXZVV7AJWWSWT46PYTPROPXK3FKT6RG6OA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.068071169 (549h54m53.939154214s) [noise=true , address=77.90.31.13:4637 , piece_id=3DXUVK5GKJTS44J7YRS4F4Q2LCJZPXR2NO33X4USDS4ZT2TRWZQQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344652626 (549h54m54.215735671s) [noise=true , address=77.90.31.18:4816 , piece_id=VFDG57SF4VBQ2QNO6G47Y3GVPKPPRXEIDSKLFPOHPBDM2IEPRSSQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: <nil> Dec 28 13:23:54.344680299 (549h54m54.215763344s) [seg_pos={0 0}, upload=482157]: Segment reached optimal threshold of 14 pieces. Dec 28 13:23:54.344956455 (549h54m54.2160395s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.344966564 (549h54m54.216049599s) [noise=true , address=145.79.227.33:4873 , piece_id=7ZA7Y4JEC46ISXCODDZWTWK5Z2PW63RQXCXHYLOZIHPNOAM7OZ5Q, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345149353 (549h54m54.216232388s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pTsKDUmbk8xYNnjrcYyX4qxwLSfHzsGgkUGSbrBiWPARfBrfi): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345159442 (549h54m54.216242497s) [noise=true , address=145.79.227.44:4208 , piece_id=IE3WJF5WBBD5QLRO6RGFB6B65MOXXJFFAQ3VNWKX5UGXCM7WFFOQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345207493 (549h54m54.216290538s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:60624->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 13:23:54.345232250 (549h54m54.216315295s) [noise=true , address=145.79.227.30:4796 , piece_id=WXIDB5AVYYQI2OLXYKOF2XIJSZXAYJYOTB42NXDTDUQF7NQ3YJXA, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345346027 (549h54m54.216429082s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T): protocol: expected piece hash; context canceled; EOF Dec 28 13:23:54.345355706 (549h54m54.216438741s) [noise=true , address=145.79.227.43:4186 , piece_id=IKTBRT7FMJ4TVBVGBWDGBX5ERVA5UMCI2WRBKWYRRNPKGNJ42JJQ, seg_pos={0 0}, upload=482157]: Long tail cancellation due to optimal threshold Dec 28 13:23:54.345380442 (549h54m54.216463477s) [seg_pos={0 0}, upload=482157]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 13:23:54.345469232 (549h54m54.216552277s) [upload=482157]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:23:54.345475804 (549h54m54.216558849s) [upload=482157]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:23:54.639252203 (549h54m54.510335248s) [upload=482157]: Done uploading segments... Dec 28 13:23:54.639277110 (549h54m54.510360165s) [upload=482157]: Upload finished. err: <nil> Duration: 32.217781191s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.X..XX.X.X.XXXXX.X.XXXXXXXXX.XXXXXXX.XX.X.XXXXXXX.XXXXXXX.XX] duration: 5.000824984s trace: https://jaeger.dev.storj.io/trace/4f42a5dcf68d2b65 FAIL (2025-12-28 13:37:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000840863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X..........X.XX..XX..X.......XXX......XX......X..X.....X..] duration: 5.97884784s trace: https://jaeger.dev.storj.io/trace/784a3f13b4812d01 FAIL (2025-12-28 13:37:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA) context deadline exceeded ======LOGS====== Dec 28 13:35:50.062665028 (550h6m49.933748073s) [upload=482321]: Starting upload Dec 28 13:35:50.062748207 (550h6m49.933831242s) [upload=482321]: Done starting upload Dec 28 13:35:50.062809193 (550h6m49.933892228s) [upload=482321]: Uploading segments... Dec 28 13:35:50.062965481 (550h6m49.934048516s) [upload=482321]: Got next segment. Inline: false Dec 28 13:35:50.063021718 (550h6m49.934104753s) [upload=482321]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 13:35:50.063026357 (550h6m49.934109392s) [upload=482321]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 13:35:50.310224221 (550h6m50.181307286s) [seg_pos={0 0}, upload=482321]: Begin upload segment... Dec 28 13:35:50.310295397 (550h6m50.181378432s) [seg_pos={0 0}, upload=482321]: Done begin upload segment. Dec 28 13:35:50.310331626 (550h6m50.181414671s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.310358276 (550h6m50.181441321s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.310419774 (550h6m50.181502829s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.310947418 (550h6m50.182030453s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.311143151 (550h6m50.182226196s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.311277658 (550h6m50.182360693s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.311831001 (550h6m50.182914046s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.312382591 (550h6m50.183465636s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.312969849 (550h6m50.184052884s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.313353800 (550h6m50.184436835s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.313726510 (550h6m50.184809555s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.313835117 (550h6m50.184918162s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.314393881 (550h6m50.185476936s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.314780888 (550h6m50.185863933s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.315126686 (550h6m50.186209731s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:50.315308213 (550h6m50.186391258s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Uploading piece... Dec 28 13:35:52.019101362 (550h6m51.890184408s) [upload=482321]: Next returned nil segment Dec 28 13:35:52.019126300 (550h6m51.890209335s) [upload=482321]: Waiting for error group managing segments... Dec 28 13:35:52.306862006 (550h6m52.177945041s) [noise=true , address=145.79.227.7:4294 , piece_id=HWQV2ADSMUEVGZBKXITESLRFNXCZFRJYSAQVCOE3OQLTCJQSNJLQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.312294029 (550h6m52.183377065s) [noise=true , address=145.79.227.46:4603 , piece_id=VAHDGYDXEVYP3YZP4EYZCZNMDOZTGMOHOHFYZSSZQ3N753GQZL6A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.316843582 (550h6m52.187926617s) [noise=true , address=145.79.227.13:4804 , piece_id=3EU4I6ROCN6M7S2HWK5R2BTQKODWRWWQUYKBRCFWUPHXLYJAG2NA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.320101154 (550h6m52.191184199s) [noise=true , address=145.79.227.48:4303 , piece_id=5BSEGH2QCS5KNZ3PGQNNDWW2OCK4GXPH65JF3UOTIAYFR565C4CA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.321281732 (550h6m52.192364767s) [noise=true , address=145.79.227.42:4121 , piece_id=UIK6ZGDT6ORFYSFG4OVDU5YM663AQYY3LDRPCATE5F3COFYCUUZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.325875530 (550h6m52.196958575s) [noise=true , address=145.79.227.47:4265 , piece_id=P5WHW55DIOFQ276XKSJYTETLPJIUVXX4G6TLJFMH6SDMB4Y67MUA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.332572743 (550h6m52.203655788s) [noise=true , address=145.79.227.39:4013 , piece_id=QAZYYCOBP6PPWQNDL5QP46HPS54VDFUR32NF6O6UYG7PPM2DOHMQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.336657431 (550h6m52.207740476s) [noise=true , address=145.79.227.37:4566 , piece_id=VSNLGTM6WTVLQKEXTKQRTOKQBU7473OXAVBLWDWWKHT7UNLO3AIQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.349638088 (550h6m52.220721133s) [noise=true , address=145.79.227.18:4007 , piece_id=WNXDUNKIIULVXHX4YYKAOLMXZJF3YR34CESTPEYYVZQRBSUA2RRA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.585376954 (550h6m52.456459999s) [noise=true , address=45.91.116.9:4264 , piece_id=VELDTZAONHZMGEQUVGWUGMUMKUESJAHRBCSEADYOL2Z357MT5H7A, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.613580307 (550h6m52.484663352s) [noise=true , address=45.91.116.8:4047 , piece_id=WCV545DMFSOXL6W45I2TQGMW334ETIJ5RKA5VZKMOEI4SXKJJKNA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.634525711 (550h6m52.505608746s) [noise=true , address=45.91.116.14:4192 , piece_id=32SUYFVMXV4RU36TMYYNX2C77TL6FL4OVTBLAIFM2RPS53GPGGZQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.679552599 (550h6m52.550635644s) [noise=true , address=45.91.116.10:4083 , piece_id=NZUWHOABYO2QMGFUQIKECRS2L4KHDLS5V5U2QBRXAIW754IOZSFA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: <nil> Dec 28 13:35:52.679570724 (550h6m52.550653759s) [seg_pos={0 0}, upload=482321]: Segment reached optimal threshold of 13 pieces. Dec 28 13:35:52.679790232 (550h6m52.550873277s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 13:35:52.679800371 (550h6m52.550883416s) [noise=true , address=77.90.31.16:4735 , piece_id=RJFLMJX26DLQXRHU35DKVPBKSHQXEXFIF55PBONAK6EERLO5M4UA, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold Dec 28 13:35:52.680032393 (550h6m52.551115428s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: expected piece hash; context canceled; EOF Dec 28 13:35:52.680073130 (550h6m52.551156165s) [noise=true , address=145.79.227.15:4873 , piece_id=OCN2SZDGM2EJW225RTHADBLKE2UJNGVQYVNJCMKIABFRUVANHQQQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold Dec 28 13:35:52.680112565 (550h6m52.5511956s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60280->45.91.116.6:4974: use of closed network connection Dec 28 13:35:52.680121052 (550h6m52.551204087s) [noise=true , address=45.91.116.6:4974 , piece_id=E2KIVDUVWGW52NJVPNFHNATFP6U2GNAKHWV2RNXIYBU42M47R7AQ, seg_pos={0 0}, upload=482321]: Long tail cancellation due to optimal threshold Dec 28 13:35:52.680140990 (550h6m52.551224025s) [seg_pos={0 0}, upload=482321]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 13:35:52.680209109 (550h6m52.551292164s) [upload=482321]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 13:35:52.680214860 (550h6m52.551297905s) [upload=482321]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 13:35:52.975175162 (550h6m52.846258197s) [upload=482321]: Done uploading segments... Dec 28 13:35:52.975223415 (550h6m52.84630645s) [upload=482321]: Upload finished. err: <nil> Duration: 5.978873007s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)