Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X.....X.........X.....X.................X.....X.X..........] duration: 3.361570118s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:12:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.36159682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X..] duration: 5.672056039s trace: https://jaeger.dev.storj.io/trace/2b1b6dbc7f2e572 FAIL (2025-12-27 07:11:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (519h41m44.57501836s) [upload=455932]: Done starting upload Dec 27 07:10:44.703982134 (519h41m44.575065169s) [upload=455932]: Uploading segments... Dec 27 07:10:44.704164070 (519h41m44.575247115s) [upload=455932]: Got next segment. Inline: false Dec 27 07:10:44.704238722 (519h41m44.575321757s) [upload=455932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:10:44.704249684 (519h41m44.575332719s) [upload=455932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:10:44.948823943 (519h41m44.819906988s) [seg_pos={0 0}, upload=455932]: Begin upload segment... Dec 27 07:10:44.948892694 (519h41m44.819975729s) [seg_pos={0 0}, upload=455932]: Done begin upload segment. Dec 27 07:10:44.948940484 (519h41m44.820023529s) [noise=true , address=145.79.227.4:4186 , piece_id=S3IXZFCBA3RS5WPEJDVNISU53NITM7PMPFP27H2VZKQDUSVSCFYA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.948949842 (519h41m44.820032887s) [noise=true , address=145.79.227.29:4941 , piece_id=4VPTWT2YBKB22LLFGRAQMNGZSZJORCPO2UW7MAGOLPUN2TY2ZLDA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.949550496 (519h41m44.820633531s) [noise=true , address=145.79.227.6:4259 , piece_id=N2VEVWFA7PB323NYVUXJ6BLKJBZSPU7PLODHQB5PEU6E6IDVQ5LQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.949695893 (519h41m44.820778928s) [noise=true , address=45.91.116.7:4009 , piece_id=3FLM5V7HXNB6U5JODOI2WJGFJIX2L3NCMHLJJ4LJUWK57Z6F5ZVQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950061589 (519h41m44.821144634s) [noise=true , address=145.79.227.15:4873 , piece_id=GXW6TXXI35STPRY5OESRAC4S7OZVR4I45TGCS7FCFWJFB42JU4KA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950421495 (519h41m44.82150453s) [noise=true , address=145.79.227.28:4727 , piece_id=7YZT755JEDE3444FP6YGMANWCIBOVKQNN64TIVHBSNJZOB4NRCSA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950708661 (519h41m44.821791706s) [noise=true , address=145.79.227.42:4122 , piece_id=3BN3QQTO5I2YE36TICSVSOZT3ZNNCMSQEUXSTYKQTM3PZNBM5KRQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950715574 (519h41m44.821798609s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951058848 (519h41m44.822141883s) [noise=true , address=145.79.227.52:4446 , piece_id=LL6ARPQGRBS6AEJJ4JSNGEUOJKIGKIWPX7UB7HH64RFWC3IQO4KQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951317210 (519h41m44.822400255s) [noise=true , address=145.79.227.22:4510 , piece_id=ARMIBBBPJBF3LYZEBHRUM5YEFUO4YUWJGC2LE4DD6M4EUN6IB7TA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951518313 (519h41m44.822601348s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDL6GDFUHRKGUYP7M26JAQNJ6TMAFRNIVEVX57YP75ZZ3ZZDDYEQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951932291 (519h41m44.823015326s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951963761 (519h41m44.823046806s) [noise=true , address=145.79.227.31:4797 , piece_id=GBPWNX6W4OWTBP3GPCXWGUTZIUWPBJTAJ5UOLSQYSFCI7PCJTJCA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.952203067 (519h41m44.823286102s) [noise=true , address=145.79.227.38:4978 , piece_id=ZJ5LMOHMNMVNA2IEOGPF4QEBSXJBBGJK6GUUDKO3OL7S4YVHXWLQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.952730932 (519h41m44.823813977s) [noise=true , address=145.79.227.11:4730 , piece_id=RFVTDGDUHVY2QLTDIYZCAC7D5HWDMZWT3NSEQGDQ725YPYZSH6AQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.953297330 (519h41m44.824380375s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:46.658561914 (519h41m46.529644959s) [upload=455932]: Next returned nil segment Dec 27 07:10:46.658576281 (519h41m46.529659326s) [upload=455932]: Waiting for error group managing segments... Dec 27 07:10:46.950163581 (519h41m46.821246626s) [noise=true , address=145.79.227.29:4941 , piece_id=4VPTWT2YBKB22LLFGRAQMNGZSZJORCPO2UW7MAGOLPUN2TY2ZLDA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.953892882 (519h41m46.824975917s) [noise=true , address=145.79.227.15:4873 , piece_id=GXW6TXXI35STPRY5OESRAC4S7OZVR4I45TGCS7FCFWJFB42JU4KA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.957328775 (519h41m46.82841181s) [noise=true , address=145.79.227.22:4510 , piece_id=ARMIBBBPJBF3LYZEBHRUM5YEFUO4YUWJGC2LE4DD6M4EUN6IB7TA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.961319113 (519h41m46.832402148s) [noise=true , address=145.79.227.11:4730 , piece_id=RFVTDGDUHVY2QLTDIYZCAC7D5HWDMZWT3NSEQGDQ725YPYZSH6AQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.961690471 (519h41m46.832773506s) [noise=true , address=145.79.227.28:4727 , piece_id=7YZT755JEDE3444FP6YGMANWCIBOVKQNN64TIVHBSNJZOB4NRCSA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.966505219 (519h41m46.837588264s) [noise=true , address=145.79.227.4:4186 , piece_id=S3IXZFCBA3RS5WPEJDVNISU53NITM7PMPFP27H2VZKQDUSVSCFYA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.973983539 (519h41m46.845066574s) [noise=true , address=145.79.227.42:4122 , piece_id=3BN3QQTO5I2YE36TICSVSOZT3ZNNCMSQEUXSTYKQTM3PZNBM5KRQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.974420662 (519h41m46.845503707s) [noise=true , address=145.79.227.38:4978 , piece_id=ZJ5LMOHMNMVNA2IEOGPF4QEBSXJBBGJK6GUUDKO3OL7S4YVHXWLQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.977131403 (519h41m46.848214438s) [noise=true , address=145.79.227.6:4259 , piece_id=N2VEVWFA7PB323NYVUXJ6BLKJBZSPU7PLODHQB5PEU6E6IDVQ5LQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.977377312 (519h41m46.848460357s) [noise=true , address=145.79.227.31:4797 , piece_id=GBPWNX6W4OWTBP3GPCXWGUTZIUWPBJTAJ5UOLSQYSFCI7PCJTJCA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.982048867 (519h41m46.853131902s) [noise=true , address=145.79.227.52:4446 , piece_id=LL6ARPQGRBS6AEJJ4JSNGEUOJKIGKIWPX7UB7HH64RFWC3IQO4KQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.278479259 (519h41m47.149562294s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDL6GDFUHRKGUYP7M26JAQNJ6TMAFRNIVEVX57YP75ZZ3ZZDDYEQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.329129307 (519h41m47.200212342s) [noise=true , address=45.91.116.7:4009 , piece_id=3FLM5V7HXNB6U5JODOI2WJGFJIX2L3NCMHLJJ4LJUWK57Z6F5ZVQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.329147321 (519h41m47.200230356s) [seg_pos={0 0}, upload=455932]: Segment reached optimal threshold of 13 pieces. Dec 27 07:10:47.329350428 (519h41m47.200433463s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection Dec 27 07:10:47.329358022 (519h41m47.200441067s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329617948 (519h41m47.200700983s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:56262->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:10:47.329624109 (519h41m47.200707154s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329644538 (519h41m47.200727573s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 07:10:47.329660038 (519h41m47.200743083s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329707508 (519h41m47.200790543s) [seg_pos={0 0}, upload=455932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:10:47.329850159 (519h41m47.200933194s) [upload=455932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:10:47.329863746 (519h41m47.200946781s) [upload=455932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:10:47.623915719 (519h41m47.494998754s) [upload=455932]: Done uploading segments... Dec 27 07:10:47.623934805 (519h41m47.49501784s) [upload=455932]: Upload finished. err: <nil> Duration: 5.672182169s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..X.....X.........X.....X.................X.....X.X.........] duration: 3.44290844s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:11:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442933447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X...................................X......X..........] duration: 42.652500643s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:11:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 42.652516253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X.] duration: 5.895188287s trace: https://jaeger.dev.storj.io/trace/2b1b6dbc7f2e572 FAIL (2025-12-27 07:11:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (519h41m44.57501836s) [upload=455932]: Done starting upload Dec 27 07:10:44.703982134 (519h41m44.575065169s) [upload=455932]: Uploading segments... Dec 27 07:10:44.704164070 (519h41m44.575247115s) [upload=455932]: Got next segment. Inline: false Dec 27 07:10:44.704238722 (519h41m44.575321757s) [upload=455932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:10:44.704249684 (519h41m44.575332719s) [upload=455932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:10:44.948823943 (519h41m44.819906988s) [seg_pos={0 0}, upload=455932]: Begin upload segment... Dec 27 07:10:44.948892694 (519h41m44.819975729s) [seg_pos={0 0}, upload=455932]: Done begin upload segment. Dec 27 07:10:44.948940484 (519h41m44.820023529s) [noise=true , address=145.79.227.4:4186 , piece_id=S3IXZFCBA3RS5WPEJDVNISU53NITM7PMPFP27H2VZKQDUSVSCFYA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.948949842 (519h41m44.820032887s) [noise=true , address=145.79.227.29:4941 , piece_id=4VPTWT2YBKB22LLFGRAQMNGZSZJORCPO2UW7MAGOLPUN2TY2ZLDA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.949550496 (519h41m44.820633531s) [noise=true , address=145.79.227.6:4259 , piece_id=N2VEVWFA7PB323NYVUXJ6BLKJBZSPU7PLODHQB5PEU6E6IDVQ5LQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.949695893 (519h41m44.820778928s) [noise=true , address=45.91.116.7:4009 , piece_id=3FLM5V7HXNB6U5JODOI2WJGFJIX2L3NCMHLJJ4LJUWK57Z6F5ZVQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950061589 (519h41m44.821144634s) [noise=true , address=145.79.227.15:4873 , piece_id=GXW6TXXI35STPRY5OESRAC4S7OZVR4I45TGCS7FCFWJFB42JU4KA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950421495 (519h41m44.82150453s) [noise=true , address=145.79.227.28:4727 , piece_id=7YZT755JEDE3444FP6YGMANWCIBOVKQNN64TIVHBSNJZOB4NRCSA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950708661 (519h41m44.821791706s) [noise=true , address=145.79.227.42:4122 , piece_id=3BN3QQTO5I2YE36TICSVSOZT3ZNNCMSQEUXSTYKQTM3PZNBM5KRQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950715574 (519h41m44.821798609s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951058848 (519h41m44.822141883s) [noise=true , address=145.79.227.52:4446 , piece_id=LL6ARPQGRBS6AEJJ4JSNGEUOJKIGKIWPX7UB7HH64RFWC3IQO4KQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951317210 (519h41m44.822400255s) [noise=true , address=145.79.227.22:4510 , piece_id=ARMIBBBPJBF3LYZEBHRUM5YEFUO4YUWJGC2LE4DD6M4EUN6IB7TA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951518313 (519h41m44.822601348s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDL6GDFUHRKGUYP7M26JAQNJ6TMAFRNIVEVX57YP75ZZ3ZZDDYEQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951932291 (519h41m44.823015326s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951963761 (519h41m44.823046806s) [noise=true , address=145.79.227.31:4797 , piece_id=GBPWNX6W4OWTBP3GPCXWGUTZIUWPBJTAJ5UOLSQYSFCI7PCJTJCA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.952203067 (519h41m44.823286102s) [noise=true , address=145.79.227.38:4978 , piece_id=ZJ5LMOHMNMVNA2IEOGPF4QEBSXJBBGJK6GUUDKO3OL7S4YVHXWLQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.952730932 (519h41m44.823813977s) [noise=true , address=145.79.227.11:4730 , piece_id=RFVTDGDUHVY2QLTDIYZCAC7D5HWDMZWT3NSEQGDQ725YPYZSH6AQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.953297330 (519h41m44.824380375s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:46.658561914 (519h41m46.529644959s) [upload=455932]: Next returned nil segment Dec 27 07:10:46.658576281 (519h41m46.529659326s) [upload=455932]: Waiting for error group managing segments... Dec 27 07:10:46.950163581 (519h41m46.821246626s) [noise=true , address=145.79.227.29:4941 , piece_id=4VPTWT2YBKB22LLFGRAQMNGZSZJORCPO2UW7MAGOLPUN2TY2ZLDA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.953892882 (519h41m46.824975917s) [noise=true , address=145.79.227.15:4873 , piece_id=GXW6TXXI35STPRY5OESRAC4S7OZVR4I45TGCS7FCFWJFB42JU4KA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.957328775 (519h41m46.82841181s) [noise=true , address=145.79.227.22:4510 , piece_id=ARMIBBBPJBF3LYZEBHRUM5YEFUO4YUWJGC2LE4DD6M4EUN6IB7TA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.961319113 (519h41m46.832402148s) [noise=true , address=145.79.227.11:4730 , piece_id=RFVTDGDUHVY2QLTDIYZCAC7D5HWDMZWT3NSEQGDQ725YPYZSH6AQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.961690471 (519h41m46.832773506s) [noise=true , address=145.79.227.28:4727 , piece_id=7YZT755JEDE3444FP6YGMANWCIBOVKQNN64TIVHBSNJZOB4NRCSA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.966505219 (519h41m46.837588264s) [noise=true , address=145.79.227.4:4186 , piece_id=S3IXZFCBA3RS5WPEJDVNISU53NITM7PMPFP27H2VZKQDUSVSCFYA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.973983539 (519h41m46.845066574s) [noise=true , address=145.79.227.42:4122 , piece_id=3BN3QQTO5I2YE36TICSVSOZT3ZNNCMSQEUXSTYKQTM3PZNBM5KRQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.974420662 (519h41m46.845503707s) [noise=true , address=145.79.227.38:4978 , piece_id=ZJ5LMOHMNMVNA2IEOGPF4QEBSXJBBGJK6GUUDKO3OL7S4YVHXWLQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.977131403 (519h41m46.848214438s) [noise=true , address=145.79.227.6:4259 , piece_id=N2VEVWFA7PB323NYVUXJ6BLKJBZSPU7PLODHQB5PEU6E6IDVQ5LQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.977377312 (519h41m46.848460357s) [noise=true , address=145.79.227.31:4797 , piece_id=GBPWNX6W4OWTBP3GPCXWGUTZIUWPBJTAJ5UOLSQYSFCI7PCJTJCA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.982048867 (519h41m46.853131902s) [noise=true , address=145.79.227.52:4446 , piece_id=LL6ARPQGRBS6AEJJ4JSNGEUOJKIGKIWPX7UB7HH64RFWC3IQO4KQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.278479259 (519h41m47.149562294s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDL6GDFUHRKGUYP7M26JAQNJ6TMAFRNIVEVX57YP75ZZ3ZZDDYEQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.329129307 (519h41m47.200212342s) [noise=true , address=45.91.116.7:4009 , piece_id=3FLM5V7HXNB6U5JODOI2WJGFJIX2L3NCMHLJJ4LJUWK57Z6F5ZVQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.329147321 (519h41m47.200230356s) [seg_pos={0 0}, upload=455932]: Segment reached optimal threshold of 13 pieces. Dec 27 07:10:47.329350428 (519h41m47.200433463s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection Dec 27 07:10:47.329358022 (519h41m47.200441067s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329617948 (519h41m47.200700983s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:56262->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:10:47.329624109 (519h41m47.200707154s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329644538 (519h41m47.200727573s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 07:10:47.329660038 (519h41m47.200743083s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329707508 (519h41m47.200790543s) [seg_pos={0 0}, upload=455932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:10:47.329850159 (519h41m47.200933194s) [upload=455932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:10:47.329863746 (519h41m47.200946781s) [upload=455932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:10:47.623915719 (519h41m47.494998754s) [upload=455932]: Done uploading segments... Dec 27 07:10:47.623934805 (519h41m47.49501784s) [upload=455932]: Upload finished. err: <nil> Duration: 5.895219516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X..X] duration: 30.00511263s trace: https://jaeger.dev.storj.io/trace/2b1b6dbc7f2e572 FAIL (2025-12-27 07:11:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (519h41m44.57501836s) [upload=455932]: Done starting upload Dec 27 07:10:44.703982134 (519h41m44.575065169s) [upload=455932]: Uploading segments... Dec 27 07:10:44.704164070 (519h41m44.575247115s) [upload=455932]: Got next segment. Inline: false Dec 27 07:10:44.704238722 (519h41m44.575321757s) [upload=455932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:10:44.704249684 (519h41m44.575332719s) [upload=455932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:10:44.948823943 (519h41m44.819906988s) [seg_pos={0 0}, upload=455932]: Begin upload segment... Dec 27 07:10:44.948892694 (519h41m44.819975729s) [seg_pos={0 0}, upload=455932]: Done begin upload segment. Dec 27 07:10:44.948940484 (519h41m44.820023529s) [noise=true , address=145.79.227.4:4186 , piece_id=S3IXZFCBA3RS5WPEJDVNISU53NITM7PMPFP27H2VZKQDUSVSCFYA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.948949842 (519h41m44.820032887s) [noise=true , address=145.79.227.29:4941 , piece_id=4VPTWT2YBKB22LLFGRAQMNGZSZJORCPO2UW7MAGOLPUN2TY2ZLDA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.949550496 (519h41m44.820633531s) [noise=true , address=145.79.227.6:4259 , piece_id=N2VEVWFA7PB323NYVUXJ6BLKJBZSPU7PLODHQB5PEU6E6IDVQ5LQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.949695893 (519h41m44.820778928s) [noise=true , address=45.91.116.7:4009 , piece_id=3FLM5V7HXNB6U5JODOI2WJGFJIX2L3NCMHLJJ4LJUWK57Z6F5ZVQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950061589 (519h41m44.821144634s) [noise=true , address=145.79.227.15:4873 , piece_id=GXW6TXXI35STPRY5OESRAC4S7OZVR4I45TGCS7FCFWJFB42JU4KA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950421495 (519h41m44.82150453s) [noise=true , address=145.79.227.28:4727 , piece_id=7YZT755JEDE3444FP6YGMANWCIBOVKQNN64TIVHBSNJZOB4NRCSA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950708661 (519h41m44.821791706s) [noise=true , address=145.79.227.42:4122 , piece_id=3BN3QQTO5I2YE36TICSVSOZT3ZNNCMSQEUXSTYKQTM3PZNBM5KRQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.950715574 (519h41m44.821798609s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951058848 (519h41m44.822141883s) [noise=true , address=145.79.227.52:4446 , piece_id=LL6ARPQGRBS6AEJJ4JSNGEUOJKIGKIWPX7UB7HH64RFWC3IQO4KQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951317210 (519h41m44.822400255s) [noise=true , address=145.79.227.22:4510 , piece_id=ARMIBBBPJBF3LYZEBHRUM5YEFUO4YUWJGC2LE4DD6M4EUN6IB7TA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951518313 (519h41m44.822601348s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDL6GDFUHRKGUYP7M26JAQNJ6TMAFRNIVEVX57YP75ZZ3ZZDDYEQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951932291 (519h41m44.823015326s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.951963761 (519h41m44.823046806s) [noise=true , address=145.79.227.31:4797 , piece_id=GBPWNX6W4OWTBP3GPCXWGUTZIUWPBJTAJ5UOLSQYSFCI7PCJTJCA, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.952203067 (519h41m44.823286102s) [noise=true , address=145.79.227.38:4978 , piece_id=ZJ5LMOHMNMVNA2IEOGPF4QEBSXJBBGJK6GUUDKO3OL7S4YVHXWLQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.952730932 (519h41m44.823813977s) [noise=true , address=145.79.227.11:4730 , piece_id=RFVTDGDUHVY2QLTDIYZCAC7D5HWDMZWT3NSEQGDQ725YPYZSH6AQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:44.953297330 (519h41m44.824380375s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Uploading piece... Dec 27 07:10:46.658561914 (519h41m46.529644959s) [upload=455932]: Next returned nil segment Dec 27 07:10:46.658576281 (519h41m46.529659326s) [upload=455932]: Waiting for error group managing segments... Dec 27 07:10:46.950163581 (519h41m46.821246626s) [noise=true , address=145.79.227.29:4941 , piece_id=4VPTWT2YBKB22LLFGRAQMNGZSZJORCPO2UW7MAGOLPUN2TY2ZLDA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.953892882 (519h41m46.824975917s) [noise=true , address=145.79.227.15:4873 , piece_id=GXW6TXXI35STPRY5OESRAC4S7OZVR4I45TGCS7FCFWJFB42JU4KA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.957328775 (519h41m46.82841181s) [noise=true , address=145.79.227.22:4510 , piece_id=ARMIBBBPJBF3LYZEBHRUM5YEFUO4YUWJGC2LE4DD6M4EUN6IB7TA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.961319113 (519h41m46.832402148s) [noise=true , address=145.79.227.11:4730 , piece_id=RFVTDGDUHVY2QLTDIYZCAC7D5HWDMZWT3NSEQGDQ725YPYZSH6AQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.961690471 (519h41m46.832773506s) [noise=true , address=145.79.227.28:4727 , piece_id=7YZT755JEDE3444FP6YGMANWCIBOVKQNN64TIVHBSNJZOB4NRCSA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.966505219 (519h41m46.837588264s) [noise=true , address=145.79.227.4:4186 , piece_id=S3IXZFCBA3RS5WPEJDVNISU53NITM7PMPFP27H2VZKQDUSVSCFYA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.973983539 (519h41m46.845066574s) [noise=true , address=145.79.227.42:4122 , piece_id=3BN3QQTO5I2YE36TICSVSOZT3ZNNCMSQEUXSTYKQTM3PZNBM5KRQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.974420662 (519h41m46.845503707s) [noise=true , address=145.79.227.38:4978 , piece_id=ZJ5LMOHMNMVNA2IEOGPF4QEBSXJBBGJK6GUUDKO3OL7S4YVHXWLQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.977131403 (519h41m46.848214438s) [noise=true , address=145.79.227.6:4259 , piece_id=N2VEVWFA7PB323NYVUXJ6BLKJBZSPU7PLODHQB5PEU6E6IDVQ5LQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.977377312 (519h41m46.848460357s) [noise=true , address=145.79.227.31:4797 , piece_id=GBPWNX6W4OWTBP3GPCXWGUTZIUWPBJTAJ5UOLSQYSFCI7PCJTJCA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:46.982048867 (519h41m46.853131902s) [noise=true , address=145.79.227.52:4446 , piece_id=LL6ARPQGRBS6AEJJ4JSNGEUOJKIGKIWPX7UB7HH64RFWC3IQO4KQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.278479259 (519h41m47.149562294s) [noise=true , address=77.90.31.11:4556 , piece_id=ZDL6GDFUHRKGUYP7M26JAQNJ6TMAFRNIVEVX57YP75ZZ3ZZDDYEQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.329129307 (519h41m47.200212342s) [noise=true , address=45.91.116.7:4009 , piece_id=3FLM5V7HXNB6U5JODOI2WJGFJIX2L3NCMHLJJ4LJUWK57Z6F5ZVQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: <nil> Dec 27 07:10:47.329147321 (519h41m47.200230356s) [seg_pos={0 0}, upload=455932]: Segment reached optimal threshold of 13 pieces. Dec 27 07:10:47.329350428 (519h41m47.200433463s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45682->45.91.116.8:4046: use of closed network connection Dec 27 07:10:47.329358022 (519h41m47.200441067s) [noise=true , address=45.91.116.8:4046 , piece_id=2C43SRDQDJUV3NV4CBYZU3CZIUCCX2FWF4OLPVQ6KCDQRM2DACNQ, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329617948 (519h41m47.200700983s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:56262->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 07:10:47.329624109 (519h41m47.200707154s) [noise=true , address=77.90.31.13:4627 , piece_id=TOUSOKS3CHVZJ5ZBVNABOFRNAEAZ6R27DKWM32O6TSQ62F3BD4OA, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329644538 (519h41m47.200727573s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 27 07:10:47.329660038 (519h41m47.200743083s) [noise=true , address=45.91.116.14:4192 , piece_id=NAXNM4XXMX5UYRHY7PVE63EUILSCDKJCGH6AGNUSR7KI5YUCD3BQ, seg_pos={0 0}, upload=455932]: Long tail cancellation due to optimal threshold Dec 27 07:10:47.329707508 (519h41m47.200790543s) [seg_pos={0 0}, upload=455932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:10:47.329850159 (519h41m47.200933194s) [upload=455932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:10:47.329863746 (519h41m47.200946781s) [upload=455932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:10:47.623915719 (519h41m47.494998754s) [upload=455932]: Done uploading segments... Dec 27 07:10:47.623934805 (519h41m47.49501784s) [upload=455932]: Upload finished. err: <nil> Duration: 30.005139751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...................................X......X.........] duration: 35.804836765s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:10:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 35.804854478s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...X.....X.........X.....X.................X.....X.X........] duration: 3.483540984s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:10:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.483606027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X..] duration: 5.624301937s trace: https://jaeger.dev.storj.io/trace/231027ade498343 FAIL (2025-12-27 07:10:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 07:08:23.388259196 (519h39m23.259342231s) [upload=455897]: Starting upload Dec 27 07:08:23.388316937 (519h39m23.259399972s) [upload=455897]: Done starting upload Dec 27 07:08:23.388409923 (519h39m23.259492958s) [upload=455897]: Uploading segments... Dec 27 07:08:23.388491409 (519h39m23.259574444s) [upload=455897]: Got next segment. Inline: false Dec 27 07:08:23.388534461 (519h39m23.259617496s) [upload=455897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:08:23.388539019 (519h39m23.259622054s) [upload=455897]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:08:23.634665192 (519h39m23.505748238s) [seg_pos={0 0}, upload=455897]: Begin upload segment... Dec 27 07:08:23.634719617 (519h39m23.505802652s) [seg_pos={0 0}, upload=455897]: Done begin upload segment. Dec 27 07:08:23.634868610 (519h39m23.505951655s) [noise=true , address=77.90.31.18:4808 , piece_id=ZMD6472Q5ZQHCHCLQAZ76NT6SWQDDU4FCUTEOH7C5HBSM3RBR4KA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.634995382 (519h39m23.506078417s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.635716254 (519h39m23.506799289s) [noise=true , address=145.79.227.22:4510 , piece_id=XNV7Z3BTUY23DJLZALW2XOVHW4VE4TK2PWJQJYRLIGON5CXCKW5Q, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.636425375 (519h39m23.50750842s) [noise=true , address=145.79.227.15:4874 , piece_id=6FOUTN4SY3A4VC4AN7RRDIIJAOXZAVPRH3RIDFIYKJIYFECCKK2Q, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.637588038 (519h39m23.508671083s) [noise=true , address=145.79.227.4:4186 , piece_id=UGDOMQRN5WPUDNXHT6S7THIAFXPDGHLSKUBAXJAX6SCZ6FWIBUVQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.639582646 (519h39m23.510665681s) [noise=true , address=145.79.227.39:4015 , piece_id=EIEGKQNYLSRRU5H3LTH5JO6JPE4EULWRGZHBWATZ63VU525S3UFA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.640695495 (519h39m23.51177853s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.641575471 (519h39m23.512658506s) [noise=true , address=145.79.227.38:4978 , piece_id=VAQ4CATQKRWZCOTCTNFF6URVX73HGONNZIHZFGYK7H5DRVABPPAA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.643017597 (519h39m23.514100642s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.644940077 (519h39m23.516023123s) [noise=true , address=145.79.227.51:4409 , piece_id=TYBPCSZKCVCMTD3LKAPOCW37OJ4NOHIUPKXIUUL5QABQ4MC5NF6A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.645464666 (519h39m23.516547701s) [noise=true , address=145.79.227.20:4439 , piece_id=2J6OGKMYR7QWDZIXAQJ3DRMQJKHX2R6QT2NYBH23WNOKFSPTWT6A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.648341314 (519h39m23.519424359s) [noise=true , address=145.79.227.11:4730 , piece_id=6A6ZJE3JZXTGSWWUIDMH2SNIEXZOJD3ZSI7KEZAZNMBY5JLBM3WA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.649385022 (519h39m23.520468057s) [noise=true , address=145.79.227.42:4121 , piece_id=76B35XHPGX4TOEBACG354LOVN52HPCXXWDQMUW5SYAPZE25KDKBQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.649790133 (519h39m23.520873179s) [noise=true , address=145.79.227.13:4803 , piece_id=WQENDYLILNG32YHGYENMFXOTF7KBXFW3GYUDOBUJYC4WDDMFBCXA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.651098945 (519h39m23.52218199s) [noise=true , address=77.90.31.15:4700 , piece_id=MI7J6Q2FRWGDFX7SGLGWANXYO2PFL6FLJCMALBEEB37YWH5TLVTQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.651228863 (519h39m23.522311908s) [noise=true , address=145.79.227.6:4259 , piece_id=Q4UJVK4I5B2SJH6HYXQVPGIQ4WFEORQJN7CJYEY2KMRIIDIAULHQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:25.372211696 (519h39m25.243294731s) [upload=455897]: Next returned nil segment Dec 27 07:08:25.372225433 (519h39m25.243308468s) [upload=455897]: Waiting for error group managing segments... Dec 27 07:08:25.656560412 (519h39m25.527643457s) [noise=true , address=145.79.227.11:4730 , piece_id=6A6ZJE3JZXTGSWWUIDMH2SNIEXZOJD3ZSI7KEZAZNMBY5JLBM3WA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.659594831 (519h39m25.530677876s) [noise=true , address=145.79.227.15:4874 , piece_id=6FOUTN4SY3A4VC4AN7RRDIIJAOXZAVPRH3RIDFIYKJIYFECCKK2Q, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.660455180 (519h39m25.531538225s) [noise=true , address=145.79.227.4:4186 , piece_id=UGDOMQRN5WPUDNXHT6S7THIAFXPDGHLSKUBAXJAX6SCZ6FWIBUVQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.662986910 (519h39m25.534069955s) [noise=true , address=145.79.227.20:4439 , piece_id=2J6OGKMYR7QWDZIXAQJ3DRMQJKHX2R6QT2NYBH23WNOKFSPTWT6A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.663406379 (519h39m25.534489414s) [noise=true , address=145.79.227.51:4409 , piece_id=TYBPCSZKCVCMTD3LKAPOCW37OJ4NOHIUPKXIUUL5QABQ4MC5NF6A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.667691318 (519h39m25.538774363s) [noise=true , address=145.79.227.22:4510 , piece_id=XNV7Z3BTUY23DJLZALW2XOVHW4VE4TK2PWJQJYRLIGON5CXCKW5Q, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.675942861 (519h39m25.547025906s) [noise=true , address=145.79.227.13:4803 , piece_id=WQENDYLILNG32YHGYENMFXOTF7KBXFW3GYUDOBUJYC4WDDMFBCXA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.683445807 (519h39m25.554528853s) [noise=true , address=145.79.227.42:4121 , piece_id=76B35XHPGX4TOEBACG354LOVN52HPCXXWDQMUW5SYAPZE25KDKBQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.686480387 (519h39m25.557563422s) [noise=true , address=145.79.227.39:4015 , piece_id=EIEGKQNYLSRRU5H3LTH5JO6JPE4EULWRGZHBWATZ63VU525S3UFA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.687068566 (519h39m25.558151611s) [noise=true , address=145.79.227.6:4259 , piece_id=Q4UJVK4I5B2SJH6HYXQVPGIQ4WFEORQJN7CJYEY2KMRIIDIAULHQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.688937966 (519h39m25.560021001s) [noise=true , address=145.79.227.38:4978 , piece_id=VAQ4CATQKRWZCOTCTNFF6URVX73HGONNZIHZFGYK7H5DRVABPPAA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857213697 (519h39m25.728296742s) [noise=true , address=77.90.31.18:4808 , piece_id=ZMD6472Q5ZQHCHCLQAZ76NT6SWQDDU4FCUTEOH7C5HBSM3RBR4KA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857275324 (519h39m25.728358369s) [noise=true , address=77.90.31.15:4700 , piece_id=MI7J6Q2FRWGDFX7SGLGWANXYO2PFL6FLJCMALBEEB37YWH5TLVTQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857302385 (519h39m25.72838542s) [seg_pos={0 0}, upload=455897]: Segment reached optimal threshold of 13 pieces. Dec 27 07:08:25.857486346 (519h39m25.728569381s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection Dec 27 07:08:25.857493389 (519h39m25.728576434s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857506855 (519h39m25.72858989s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 07:08:25.857514931 (519h39m25.728597966s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857706595 (519h39m25.72878963s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 07:08:25.857712727 (519h39m25.728795772s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857729158 (519h39m25.728812194s) [seg_pos={0 0}, upload=455897]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:08:25.857852763 (519h39m25.728935808s) [upload=455897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:08:25.857857963 (519h39m25.728941008s) [upload=455897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:08:26.151481461 (519h39m26.022564506s) [upload=455897]: Done uploading segments... Dec 27 07:08:26.151505035 (519h39m26.02258808s) [upload=455897]: Upload finished. err: <nil> Duration: 5.62432938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...................................X......X........] duration: 46.290968402s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:10:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 46.290997947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X.....X.........X.....X.................X.....X.X.......] duration: 3.741369047s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:09:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.741442587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X.] duration: 5.798725715s trace: https://jaeger.dev.storj.io/trace/231027ade498343 FAIL (2025-12-27 07:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 07:08:23.388259196 (519h39m23.259342231s) [upload=455897]: Starting upload Dec 27 07:08:23.388316937 (519h39m23.259399972s) [upload=455897]: Done starting upload Dec 27 07:08:23.388409923 (519h39m23.259492958s) [upload=455897]: Uploading segments... Dec 27 07:08:23.388491409 (519h39m23.259574444s) [upload=455897]: Got next segment. Inline: false Dec 27 07:08:23.388534461 (519h39m23.259617496s) [upload=455897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:08:23.388539019 (519h39m23.259622054s) [upload=455897]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:08:23.634665192 (519h39m23.505748238s) [seg_pos={0 0}, upload=455897]: Begin upload segment... Dec 27 07:08:23.634719617 (519h39m23.505802652s) [seg_pos={0 0}, upload=455897]: Done begin upload segment. Dec 27 07:08:23.634868610 (519h39m23.505951655s) [noise=true , address=77.90.31.18:4808 , piece_id=ZMD6472Q5ZQHCHCLQAZ76NT6SWQDDU4FCUTEOH7C5HBSM3RBR4KA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.634995382 (519h39m23.506078417s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.635716254 (519h39m23.506799289s) [noise=true , address=145.79.227.22:4510 , piece_id=XNV7Z3BTUY23DJLZALW2XOVHW4VE4TK2PWJQJYRLIGON5CXCKW5Q, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.636425375 (519h39m23.50750842s) [noise=true , address=145.79.227.15:4874 , piece_id=6FOUTN4SY3A4VC4AN7RRDIIJAOXZAVPRH3RIDFIYKJIYFECCKK2Q, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.637588038 (519h39m23.508671083s) [noise=true , address=145.79.227.4:4186 , piece_id=UGDOMQRN5WPUDNXHT6S7THIAFXPDGHLSKUBAXJAX6SCZ6FWIBUVQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.639582646 (519h39m23.510665681s) [noise=true , address=145.79.227.39:4015 , piece_id=EIEGKQNYLSRRU5H3LTH5JO6JPE4EULWRGZHBWATZ63VU525S3UFA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.640695495 (519h39m23.51177853s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.641575471 (519h39m23.512658506s) [noise=true , address=145.79.227.38:4978 , piece_id=VAQ4CATQKRWZCOTCTNFF6URVX73HGONNZIHZFGYK7H5DRVABPPAA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.643017597 (519h39m23.514100642s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.644940077 (519h39m23.516023123s) [noise=true , address=145.79.227.51:4409 , piece_id=TYBPCSZKCVCMTD3LKAPOCW37OJ4NOHIUPKXIUUL5QABQ4MC5NF6A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.645464666 (519h39m23.516547701s) [noise=true , address=145.79.227.20:4439 , piece_id=2J6OGKMYR7QWDZIXAQJ3DRMQJKHX2R6QT2NYBH23WNOKFSPTWT6A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.648341314 (519h39m23.519424359s) [noise=true , address=145.79.227.11:4730 , piece_id=6A6ZJE3JZXTGSWWUIDMH2SNIEXZOJD3ZSI7KEZAZNMBY5JLBM3WA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.649385022 (519h39m23.520468057s) [noise=true , address=145.79.227.42:4121 , piece_id=76B35XHPGX4TOEBACG354LOVN52HPCXXWDQMUW5SYAPZE25KDKBQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.649790133 (519h39m23.520873179s) [noise=true , address=145.79.227.13:4803 , piece_id=WQENDYLILNG32YHGYENMFXOTF7KBXFW3GYUDOBUJYC4WDDMFBCXA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.651098945 (519h39m23.52218199s) [noise=true , address=77.90.31.15:4700 , piece_id=MI7J6Q2FRWGDFX7SGLGWANXYO2PFL6FLJCMALBEEB37YWH5TLVTQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.651228863 (519h39m23.522311908s) [noise=true , address=145.79.227.6:4259 , piece_id=Q4UJVK4I5B2SJH6HYXQVPGIQ4WFEORQJN7CJYEY2KMRIIDIAULHQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:25.372211696 (519h39m25.243294731s) [upload=455897]: Next returned nil segment Dec 27 07:08:25.372225433 (519h39m25.243308468s) [upload=455897]: Waiting for error group managing segments... Dec 27 07:08:25.656560412 (519h39m25.527643457s) [noise=true , address=145.79.227.11:4730 , piece_id=6A6ZJE3JZXTGSWWUIDMH2SNIEXZOJD3ZSI7KEZAZNMBY5JLBM3WA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.659594831 (519h39m25.530677876s) [noise=true , address=145.79.227.15:4874 , piece_id=6FOUTN4SY3A4VC4AN7RRDIIJAOXZAVPRH3RIDFIYKJIYFECCKK2Q, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.660455180 (519h39m25.531538225s) [noise=true , address=145.79.227.4:4186 , piece_id=UGDOMQRN5WPUDNXHT6S7THIAFXPDGHLSKUBAXJAX6SCZ6FWIBUVQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.662986910 (519h39m25.534069955s) [noise=true , address=145.79.227.20:4439 , piece_id=2J6OGKMYR7QWDZIXAQJ3DRMQJKHX2R6QT2NYBH23WNOKFSPTWT6A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.663406379 (519h39m25.534489414s) [noise=true , address=145.79.227.51:4409 , piece_id=TYBPCSZKCVCMTD3LKAPOCW37OJ4NOHIUPKXIUUL5QABQ4MC5NF6A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.667691318 (519h39m25.538774363s) [noise=true , address=145.79.227.22:4510 , piece_id=XNV7Z3BTUY23DJLZALW2XOVHW4VE4TK2PWJQJYRLIGON5CXCKW5Q, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.675942861 (519h39m25.547025906s) [noise=true , address=145.79.227.13:4803 , piece_id=WQENDYLILNG32YHGYENMFXOTF7KBXFW3GYUDOBUJYC4WDDMFBCXA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.683445807 (519h39m25.554528853s) [noise=true , address=145.79.227.42:4121 , piece_id=76B35XHPGX4TOEBACG354LOVN52HPCXXWDQMUW5SYAPZE25KDKBQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.686480387 (519h39m25.557563422s) [noise=true , address=145.79.227.39:4015 , piece_id=EIEGKQNYLSRRU5H3LTH5JO6JPE4EULWRGZHBWATZ63VU525S3UFA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.687068566 (519h39m25.558151611s) [noise=true , address=145.79.227.6:4259 , piece_id=Q4UJVK4I5B2SJH6HYXQVPGIQ4WFEORQJN7CJYEY2KMRIIDIAULHQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.688937966 (519h39m25.560021001s) [noise=true , address=145.79.227.38:4978 , piece_id=VAQ4CATQKRWZCOTCTNFF6URVX73HGONNZIHZFGYK7H5DRVABPPAA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857213697 (519h39m25.728296742s) [noise=true , address=77.90.31.18:4808 , piece_id=ZMD6472Q5ZQHCHCLQAZ76NT6SWQDDU4FCUTEOH7C5HBSM3RBR4KA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857275324 (519h39m25.728358369s) [noise=true , address=77.90.31.15:4700 , piece_id=MI7J6Q2FRWGDFX7SGLGWANXYO2PFL6FLJCMALBEEB37YWH5TLVTQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857302385 (519h39m25.72838542s) [seg_pos={0 0}, upload=455897]: Segment reached optimal threshold of 13 pieces. Dec 27 07:08:25.857486346 (519h39m25.728569381s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection Dec 27 07:08:25.857493389 (519h39m25.728576434s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857506855 (519h39m25.72858989s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 07:08:25.857514931 (519h39m25.728597966s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857706595 (519h39m25.72878963s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 07:08:25.857712727 (519h39m25.728795772s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857729158 (519h39m25.728812194s) [seg_pos={0 0}, upload=455897]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:08:25.857852763 (519h39m25.728935808s) [upload=455897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:08:25.857857963 (519h39m25.728941008s) [upload=455897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:08:26.151481461 (519h39m26.022564506s) [upload=455897]: Done uploading segments... Dec 27 07:08:26.151505035 (519h39m26.02258808s) [upload=455897]: Upload finished. err: <nil> Duration: 5.798756403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...................................X......X.......] duration: 33.138628089s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:09:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 33.138671732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X.....X.........X.....X.................X.....X.X......] duration: 3.502170557s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:08:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502226234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.X] duration: 30.003667793s trace: https://jaeger.dev.storj.io/trace/231027ade498343 FAIL (2025-12-27 07:08:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 07:08:23.388259196 (519h39m23.259342231s) [upload=455897]: Starting upload Dec 27 07:08:23.388316937 (519h39m23.259399972s) [upload=455897]: Done starting upload Dec 27 07:08:23.388409923 (519h39m23.259492958s) [upload=455897]: Uploading segments... Dec 27 07:08:23.388491409 (519h39m23.259574444s) [upload=455897]: Got next segment. Inline: false Dec 27 07:08:23.388534461 (519h39m23.259617496s) [upload=455897]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:08:23.388539019 (519h39m23.259622054s) [upload=455897]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:08:23.634665192 (519h39m23.505748238s) [seg_pos={0 0}, upload=455897]: Begin upload segment... Dec 27 07:08:23.634719617 (519h39m23.505802652s) [seg_pos={0 0}, upload=455897]: Done begin upload segment. Dec 27 07:08:23.634868610 (519h39m23.505951655s) [noise=true , address=77.90.31.18:4808 , piece_id=ZMD6472Q5ZQHCHCLQAZ76NT6SWQDDU4FCUTEOH7C5HBSM3RBR4KA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.634995382 (519h39m23.506078417s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.635716254 (519h39m23.506799289s) [noise=true , address=145.79.227.22:4510 , piece_id=XNV7Z3BTUY23DJLZALW2XOVHW4VE4TK2PWJQJYRLIGON5CXCKW5Q, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.636425375 (519h39m23.50750842s) [noise=true , address=145.79.227.15:4874 , piece_id=6FOUTN4SY3A4VC4AN7RRDIIJAOXZAVPRH3RIDFIYKJIYFECCKK2Q, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.637588038 (519h39m23.508671083s) [noise=true , address=145.79.227.4:4186 , piece_id=UGDOMQRN5WPUDNXHT6S7THIAFXPDGHLSKUBAXJAX6SCZ6FWIBUVQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.639582646 (519h39m23.510665681s) [noise=true , address=145.79.227.39:4015 , piece_id=EIEGKQNYLSRRU5H3LTH5JO6JPE4EULWRGZHBWATZ63VU525S3UFA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.640695495 (519h39m23.51177853s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.641575471 (519h39m23.512658506s) [noise=true , address=145.79.227.38:4978 , piece_id=VAQ4CATQKRWZCOTCTNFF6URVX73HGONNZIHZFGYK7H5DRVABPPAA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.643017597 (519h39m23.514100642s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.644940077 (519h39m23.516023123s) [noise=true , address=145.79.227.51:4409 , piece_id=TYBPCSZKCVCMTD3LKAPOCW37OJ4NOHIUPKXIUUL5QABQ4MC5NF6A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.645464666 (519h39m23.516547701s) [noise=true , address=145.79.227.20:4439 , piece_id=2J6OGKMYR7QWDZIXAQJ3DRMQJKHX2R6QT2NYBH23WNOKFSPTWT6A, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.648341314 (519h39m23.519424359s) [noise=true , address=145.79.227.11:4730 , piece_id=6A6ZJE3JZXTGSWWUIDMH2SNIEXZOJD3ZSI7KEZAZNMBY5JLBM3WA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.649385022 (519h39m23.520468057s) [noise=true , address=145.79.227.42:4121 , piece_id=76B35XHPGX4TOEBACG354LOVN52HPCXXWDQMUW5SYAPZE25KDKBQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.649790133 (519h39m23.520873179s) [noise=true , address=145.79.227.13:4803 , piece_id=WQENDYLILNG32YHGYENMFXOTF7KBXFW3GYUDOBUJYC4WDDMFBCXA, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.651098945 (519h39m23.52218199s) [noise=true , address=77.90.31.15:4700 , piece_id=MI7J6Q2FRWGDFX7SGLGWANXYO2PFL6FLJCMALBEEB37YWH5TLVTQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:23.651228863 (519h39m23.522311908s) [noise=true , address=145.79.227.6:4259 , piece_id=Q4UJVK4I5B2SJH6HYXQVPGIQ4WFEORQJN7CJYEY2KMRIIDIAULHQ, seg_pos={0 0}, upload=455897]: Uploading piece... Dec 27 07:08:25.372211696 (519h39m25.243294731s) [upload=455897]: Next returned nil segment Dec 27 07:08:25.372225433 (519h39m25.243308468s) [upload=455897]: Waiting for error group managing segments... Dec 27 07:08:25.656560412 (519h39m25.527643457s) [noise=true , address=145.79.227.11:4730 , piece_id=6A6ZJE3JZXTGSWWUIDMH2SNIEXZOJD3ZSI7KEZAZNMBY5JLBM3WA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.659594831 (519h39m25.530677876s) [noise=true , address=145.79.227.15:4874 , piece_id=6FOUTN4SY3A4VC4AN7RRDIIJAOXZAVPRH3RIDFIYKJIYFECCKK2Q, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.660455180 (519h39m25.531538225s) [noise=true , address=145.79.227.4:4186 , piece_id=UGDOMQRN5WPUDNXHT6S7THIAFXPDGHLSKUBAXJAX6SCZ6FWIBUVQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.662986910 (519h39m25.534069955s) [noise=true , address=145.79.227.20:4439 , piece_id=2J6OGKMYR7QWDZIXAQJ3DRMQJKHX2R6QT2NYBH23WNOKFSPTWT6A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.663406379 (519h39m25.534489414s) [noise=true , address=145.79.227.51:4409 , piece_id=TYBPCSZKCVCMTD3LKAPOCW37OJ4NOHIUPKXIUUL5QABQ4MC5NF6A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.667691318 (519h39m25.538774363s) [noise=true , address=145.79.227.22:4510 , piece_id=XNV7Z3BTUY23DJLZALW2XOVHW4VE4TK2PWJQJYRLIGON5CXCKW5Q, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.675942861 (519h39m25.547025906s) [noise=true , address=145.79.227.13:4803 , piece_id=WQENDYLILNG32YHGYENMFXOTF7KBXFW3GYUDOBUJYC4WDDMFBCXA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.683445807 (519h39m25.554528853s) [noise=true , address=145.79.227.42:4121 , piece_id=76B35XHPGX4TOEBACG354LOVN52HPCXXWDQMUW5SYAPZE25KDKBQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.686480387 (519h39m25.557563422s) [noise=true , address=145.79.227.39:4015 , piece_id=EIEGKQNYLSRRU5H3LTH5JO6JPE4EULWRGZHBWATZ63VU525S3UFA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.687068566 (519h39m25.558151611s) [noise=true , address=145.79.227.6:4259 , piece_id=Q4UJVK4I5B2SJH6HYXQVPGIQ4WFEORQJN7CJYEY2KMRIIDIAULHQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.688937966 (519h39m25.560021001s) [noise=true , address=145.79.227.38:4978 , piece_id=VAQ4CATQKRWZCOTCTNFF6URVX73HGONNZIHZFGYK7H5DRVABPPAA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857213697 (519h39m25.728296742s) [noise=true , address=77.90.31.18:4808 , piece_id=ZMD6472Q5ZQHCHCLQAZ76NT6SWQDDU4FCUTEOH7C5HBSM3RBR4KA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857275324 (519h39m25.728358369s) [noise=true , address=77.90.31.15:4700 , piece_id=MI7J6Q2FRWGDFX7SGLGWANXYO2PFL6FLJCMALBEEB37YWH5TLVTQ, seg_pos={0 0}, upload=455897]: Done uploading piece. err: <nil> Dec 27 07:08:25.857302385 (519h39m25.72838542s) [seg_pos={0 0}, upload=455897]: Segment reached optimal threshold of 13 pieces. Dec 27 07:08:25.857486346 (519h39m25.728569381s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47286->45.91.116.11:4118: use of closed network connection Dec 27 07:08:25.857493389 (519h39m25.728576434s) [noise=true , address=45.91.116.11:4118 , piece_id=HBZCI5EIWMQRPM4RMFKIVFSDKHZ4OMMKINAI4KWCXQ24DT2S535A, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857506855 (519h39m25.72858989s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 27 07:08:25.857514931 (519h39m25.728597966s) [noise=true , address=77.90.31.13:4627 , piece_id=OV6LC2TNWB5A5UENVWTHOG3MW6NDRJ5ENMFWNJDVJK55AIB7F6CA, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857706595 (519h39m25.72878963s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 07:08:25.857712727 (519h39m25.728795772s) [noise=true , address=45.91.116.5:4298 , piece_id=N7K2FMFIKI5AXOTKXI3GLTDX7XZH4YMS7QIZX3JQLW47U2X3GVNA, seg_pos={0 0}, upload=455897]: Long tail cancellation due to optimal threshold Dec 27 07:08:25.857729158 (519h39m25.728812194s) [seg_pos={0 0}, upload=455897]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:08:25.857852763 (519h39m25.728935808s) [upload=455897]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:08:25.857857963 (519h39m25.728941008s) [upload=455897]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:08:26.151481461 (519h39m26.022564506s) [upload=455897]: Done uploading segments... Dec 27 07:08:26.151505035 (519h39m26.02258808s) [upload=455897]: Upload finished. err: <nil> Duration: 30.003684885s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X.....X.........X.....X.................X.....X.X.....] duration: 3.457737638s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:08:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45776406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...................................X......X......] duration: 55.060850848s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 55.060866018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......X.....X.........X.....X.................X.....X.X....] duration: 3.346538825s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:08:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.346566649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X.] duration: 5.881648277s trace: https://jaeger.dev.storj.io/trace/1fe824d47cd0c7ff FAIL (2025-12-27 07:07:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: Y5KAOQER7H57ZALE57MJ4HHLMMKECYZ3A4526BENXBKXIPOBQUCA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GABYQIUFWY6AVNMAC6NA7TPQADQ36SCCBR4IPX4ASDTPQXSJUNHA) context deadline exceeded ======LOGS====== Dec 27 07:07:23.726958482 (519h38m23.598041527s) [upload=455879]: Starting upload Dec 27 07:07:23.727032023 (519h38m23.598115068s) [upload=455879]: Done starting upload Dec 27 07:07:23.727079123 (519h38m23.598162168s) [upload=455879]: Uploading segments... Dec 27 07:07:23.727400003 (519h38m23.598483049s) [upload=455879]: Got next segment. Inline: false Dec 27 07:07:23.727440931 (519h38m23.598523976s) [upload=455879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:07:23.727447585 (519h38m23.59853064s) [upload=455879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:07:23.976792676 (519h38m23.847875731s) [seg_pos={0 0}, upload=455879]: Begin upload segment... Dec 27 07:07:23.976841309 (519h38m23.847924344s) [seg_pos={0 0}, upload=455879]: Done begin upload segment. Dec 27 07:07:23.976876676 (519h38m23.847959721s) [noise=true , address=145.79.227.17:4972 , piece_id=SFRLUXDZQG22FQ4I3YJUGDDPEUY7D73OKS5E3ZZHLCPEXZRXJHCQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.976947201 (519h38m23.848030246s) [noise=true , address=77.90.31.17:4770 , piece_id=Y5KAOQER7H57ZALE57MJ4HHLMMKECYZ3A4526BENXBKXIPOBQUCA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.977364024 (519h38m23.848447059s) [noise=true , address=45.91.116.13:4156 , piece_id=GABYQIUFWY6AVNMAC6NA7TPQADQ36SCCBR4IPX4ASDTPQXSJUNHA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.977848066 (519h38m23.848931111s) [noise=true , address=145.79.227.51:4410 , piece_id=HKYBCHCG473W4AA5FCG6I7GTRHSVX4UFZ3CRSSBUZ4BIAL3ROEFQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.977871380 (519h38m23.848954415s) [noise=true , address=77.90.31.13:4629 , piece_id=4F5KH3Q3H7G3SJ2XNSFI56ZYF7HDT3SXFP54BWNC63Q656ASDY3A, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978183084 (519h38m23.849266119s) [noise=true , address=145.79.227.37:4566 , piece_id=WIX6RFI735XEZG3X4GQGYT5ORT5PGGLV5YDYIHLZE74QGM3JHHSA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978210025 (519h38m23.84929306s) [noise=true , address=77.90.31.16:4736 , piece_id=2KXPCMYZMRUOYJ54ZR6NX4UDLWEBVTXTVFGBAASWTB4WDQZG4KDA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978507893 (519h38m23.849590938s) [noise=true , address=145.79.227.25:4617 , piece_id=XHWXF3543XZZJ4PJBCFBRWSY25SEOB7BAH6XXAKBX3F6HS5SE2DQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978624484 (519h38m23.849707519s) [noise=true , address=145.79.227.19:4079 , piece_id=NNJIU3TF5372X7U672NZWHUDMYA6IMQJHPLZNDB6CTX5BAQFX6AA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978829024 (519h38m23.849912059s) [noise=true , address=45.91.116.8:4046 , piece_id=FDWTASV35EHTNDCXM27TZX5VI3I5GZ465AHQWGBEOEJVPLNSX35Q, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979044564 (519h38m23.850127599s) [noise=true , address=45.91.116.6:4973 , piece_id=5JTZRITUBOVJQ3I2Z6UOIS24D6O3TIJVWEYXRH4LJ6NMJNZCSJYA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979159853 (519h38m23.850242888s) [noise=true , address=45.91.116.10:4082 , piece_id=4INVMH7SCMZ2EADP5NCVYSJGRA3NDNT7GVBJPC5KDNPKEQI3X5EA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979457841 (519h38m23.850540876s) [noise=true , address=145.79.227.38:4978 , piece_id=XGGBDAD5AZUULI6KNCFLUKIR6GSUAULRWKOBYWEEPV4F3AFVTBYQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979467369 (519h38m23.850550404s) [noise=true , address=145.79.227.16:4909 , piece_id=DKOSLYAED5I2SCRH7B3S7MEOWOSS56F2IM2U357EWWCIHQ6ERAZQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979850679 (519h38m23.850933714s) [noise=true , address=145.79.227.39:4015 , piece_id=NX2RWBC6N4ZNQW54TL3U6CBTIOGFXHC2PWD5ENCBBNF2CY7T7LYA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.980063133 (519h38m23.851146179s) [noise=true , address=145.79.227.3:4149 , piece_id=UOMHON66HD2SFIPXSJTOKYYY67GZCYMDDDKFYH3BBRBELMD3RQKQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:25.686779951 (519h38m25.557862996s) [upload=455879]: Next returned nil segment Dec 27 07:07:25.686797474 (519h38m25.557880509s) [upload=455879]: Waiting for error group managing segments... Dec 27 07:07:25.969457967 (519h38m25.840541002s) [noise=true , address=145.79.227.37:4566 , piece_id=WIX6RFI735XEZG3X4GQGYT5ORT5PGGLV5YDYIHLZE74QGM3JHHSA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.983989226 (519h38m25.855072261s) [noise=true , address=145.79.227.38:4978 , piece_id=XGGBDAD5AZUULI6KNCFLUKIR6GSUAULRWKOBYWEEPV4F3AFVTBYQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.984860385 (519h38m25.85594343s) [noise=true , address=145.79.227.51:4410 , piece_id=HKYBCHCG473W4AA5FCG6I7GTRHSVX4UFZ3CRSSBUZ4BIAL3ROEFQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.987339887 (519h38m25.858422932s) [noise=true , address=145.79.227.17:4972 , piece_id=SFRLUXDZQG22FQ4I3YJUGDDPEUY7D73OKS5E3ZZHLCPEXZRXJHCQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.990430903 (519h38m25.861513948s) [noise=true , address=145.79.227.25:4617 , piece_id=XHWXF3543XZZJ4PJBCFBRWSY25SEOB7BAH6XXAKBX3F6HS5SE2DQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.006547252 (519h38m25.877630297s) [noise=true , address=145.79.227.19:4079 , piece_id=NNJIU3TF5372X7U672NZWHUDMYA6IMQJHPLZNDB6CTX5BAQFX6AA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.012076721 (519h38m25.883159766s) [noise=true , address=145.79.227.39:4015 , piece_id=NX2RWBC6N4ZNQW54TL3U6CBTIOGFXHC2PWD5ENCBBNF2CY7T7LYA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.014708302 (519h38m25.885791347s) [noise=true , address=145.79.227.3:4149 , piece_id=UOMHON66HD2SFIPXSJTOKYYY67GZCYMDDDKFYH3BBRBELMD3RQKQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.015276093 (519h38m25.886359128s) [noise=true , address=145.79.227.16:4909 , piece_id=DKOSLYAED5I2SCRH7B3S7MEOWOSS56F2IM2U357EWWCIHQ6ERAZQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.268670957 (519h38m26.139753992s) [noise=true , address=45.91.116.8:4046 , piece_id=FDWTASV35EHTNDCXM27TZX5VI3I5GZ465AHQWGBEOEJVPLNSX35Q, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.308284760 (519h38m26.179367795s) [noise=true , address=77.90.31.17:4770 , piece_id=Y5KAOQER7H57ZALE57MJ4HHLMMKECYZ3A4526BENXBKXIPOBQUCA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.386027277 (519h38m26.257110312s) [noise=true , address=45.91.116.10:4082 , piece_id=4INVMH7SCMZ2EADP5NCVYSJGRA3NDNT7GVBJPC5KDNPKEQI3X5EA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.436860554 (519h38m26.307943599s) [noise=true , address=45.91.116.13:4156 , piece_id=GABYQIUFWY6AVNMAC6NA7TPQADQ36SCCBR4IPX4ASDTPQXSJUNHA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.436880010 (519h38m26.307963045s) [seg_pos={0 0}, upload=455879]: Segment reached optimal threshold of 13 pieces. Dec 27 07:07:26.437043993 (519h38m26.308127028s) [noise=true , address=77.90.31.16:4736 , piece_id=2KXPCMYZMRUOYJ54ZR6NX4UDLWEBVTXTVFGBAASWTB4WDQZG4KDA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 07:07:26.437051838 (519h38m26.308134873s) [noise=true , address=77.90.31.16:4736 , piece_id=2KXPCMYZMRUOYJ54ZR6NX4UDLWEBVTXTVFGBAASWTB4WDQZG4KDA, seg_pos={0 0}, upload=455879]: Long tail cancellation due to optimal threshold Dec 27 07:07:26.437297125 (519h38m26.30838017s) [noise=true , address=77.90.31.13:4629 , piece_id=4F5KH3Q3H7G3SJ2XNSFI56ZYF7HDT3SXFP54BWNC63Q656ASDY3A, seg_pos={0 0}, upload=455879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 07:07:26.437303206 (519h38m26.308386241s) [noise=true , address=77.90.31.13:4629 , piece_id=4F5KH3Q3H7G3SJ2XNSFI56ZYF7HDT3SXFP54BWNC63Q656ASDY3A, seg_pos={0 0}, upload=455879]: Long tail cancellation due to optimal threshold Dec 27 07:07:26.437435017 (519h38m26.308518052s) [noise=true , address=45.91.116.6:4973 , piece_id=5JTZRITUBOVJQ3I2Z6UOIS24D6O3TIJVWEYXRH4LJ6NMJNZCSJYA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 07:07:26.437440528 (519h38m26.308523563s) [noise=true , address=45.91.116.6:4973 , piece_id=5JTZRITUBOVJQ3I2Z6UOIS24D6O3TIJVWEYXRH4LJ6NMJNZCSJYA, seg_pos={0 0}, upload=455879]: Long tail cancellation due to optimal threshold Dec 27 07:07:26.437474031 (519h38m26.308557066s) [seg_pos={0 0}, upload=455879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:07:26.437539256 (519h38m26.308622301s) [upload=455879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:07:26.437543504 (519h38m26.308626549s) [upload=455879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:07:26.724246193 (519h38m26.595329258s) [upload=455879]: Done uploading segments... Dec 27 07:07:26.724269558 (519h38m26.595352613s) [upload=455879]: Upload finished. err: <nil> Duration: 5.881666691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....X] duration: 30.003791833s trace: https://jaeger.dev.storj.io/trace/1fe824d47cd0c7ff FAIL (2025-12-27 07:07:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: Y5KAOQER7H57ZALE57MJ4HHLMMKECYZ3A4526BENXBKXIPOBQUCA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GABYQIUFWY6AVNMAC6NA7TPQADQ36SCCBR4IPX4ASDTPQXSJUNHA) context deadline exceeded ======LOGS====== Dec 27 07:07:23.726958482 (519h38m23.598041527s) [upload=455879]: Starting upload Dec 27 07:07:23.727032023 (519h38m23.598115068s) [upload=455879]: Done starting upload Dec 27 07:07:23.727079123 (519h38m23.598162168s) [upload=455879]: Uploading segments... Dec 27 07:07:23.727400003 (519h38m23.598483049s) [upload=455879]: Got next segment. Inline: false Dec 27 07:07:23.727440931 (519h38m23.598523976s) [upload=455879]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:07:23.727447585 (519h38m23.59853064s) [upload=455879]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:07:23.976792676 (519h38m23.847875731s) [seg_pos={0 0}, upload=455879]: Begin upload segment... Dec 27 07:07:23.976841309 (519h38m23.847924344s) [seg_pos={0 0}, upload=455879]: Done begin upload segment. Dec 27 07:07:23.976876676 (519h38m23.847959721s) [noise=true , address=145.79.227.17:4972 , piece_id=SFRLUXDZQG22FQ4I3YJUGDDPEUY7D73OKS5E3ZZHLCPEXZRXJHCQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.976947201 (519h38m23.848030246s) [noise=true , address=77.90.31.17:4770 , piece_id=Y5KAOQER7H57ZALE57MJ4HHLMMKECYZ3A4526BENXBKXIPOBQUCA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.977364024 (519h38m23.848447059s) [noise=true , address=45.91.116.13:4156 , piece_id=GABYQIUFWY6AVNMAC6NA7TPQADQ36SCCBR4IPX4ASDTPQXSJUNHA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.977848066 (519h38m23.848931111s) [noise=true , address=145.79.227.51:4410 , piece_id=HKYBCHCG473W4AA5FCG6I7GTRHSVX4UFZ3CRSSBUZ4BIAL3ROEFQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.977871380 (519h38m23.848954415s) [noise=true , address=77.90.31.13:4629 , piece_id=4F5KH3Q3H7G3SJ2XNSFI56ZYF7HDT3SXFP54BWNC63Q656ASDY3A, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978183084 (519h38m23.849266119s) [noise=true , address=145.79.227.37:4566 , piece_id=WIX6RFI735XEZG3X4GQGYT5ORT5PGGLV5YDYIHLZE74QGM3JHHSA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978210025 (519h38m23.84929306s) [noise=true , address=77.90.31.16:4736 , piece_id=2KXPCMYZMRUOYJ54ZR6NX4UDLWEBVTXTVFGBAASWTB4WDQZG4KDA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978507893 (519h38m23.849590938s) [noise=true , address=145.79.227.25:4617 , piece_id=XHWXF3543XZZJ4PJBCFBRWSY25SEOB7BAH6XXAKBX3F6HS5SE2DQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978624484 (519h38m23.849707519s) [noise=true , address=145.79.227.19:4079 , piece_id=NNJIU3TF5372X7U672NZWHUDMYA6IMQJHPLZNDB6CTX5BAQFX6AA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.978829024 (519h38m23.849912059s) [noise=true , address=45.91.116.8:4046 , piece_id=FDWTASV35EHTNDCXM27TZX5VI3I5GZ465AHQWGBEOEJVPLNSX35Q, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979044564 (519h38m23.850127599s) [noise=true , address=45.91.116.6:4973 , piece_id=5JTZRITUBOVJQ3I2Z6UOIS24D6O3TIJVWEYXRH4LJ6NMJNZCSJYA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979159853 (519h38m23.850242888s) [noise=true , address=45.91.116.10:4082 , piece_id=4INVMH7SCMZ2EADP5NCVYSJGRA3NDNT7GVBJPC5KDNPKEQI3X5EA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979457841 (519h38m23.850540876s) [noise=true , address=145.79.227.38:4978 , piece_id=XGGBDAD5AZUULI6KNCFLUKIR6GSUAULRWKOBYWEEPV4F3AFVTBYQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979467369 (519h38m23.850550404s) [noise=true , address=145.79.227.16:4909 , piece_id=DKOSLYAED5I2SCRH7B3S7MEOWOSS56F2IM2U357EWWCIHQ6ERAZQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.979850679 (519h38m23.850933714s) [noise=true , address=145.79.227.39:4015 , piece_id=NX2RWBC6N4ZNQW54TL3U6CBTIOGFXHC2PWD5ENCBBNF2CY7T7LYA, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:23.980063133 (519h38m23.851146179s) [noise=true , address=145.79.227.3:4149 , piece_id=UOMHON66HD2SFIPXSJTOKYYY67GZCYMDDDKFYH3BBRBELMD3RQKQ, seg_pos={0 0}, upload=455879]: Uploading piece... Dec 27 07:07:25.686779951 (519h38m25.557862996s) [upload=455879]: Next returned nil segment Dec 27 07:07:25.686797474 (519h38m25.557880509s) [upload=455879]: Waiting for error group managing segments... Dec 27 07:07:25.969457967 (519h38m25.840541002s) [noise=true , address=145.79.227.37:4566 , piece_id=WIX6RFI735XEZG3X4GQGYT5ORT5PGGLV5YDYIHLZE74QGM3JHHSA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.983989226 (519h38m25.855072261s) [noise=true , address=145.79.227.38:4978 , piece_id=XGGBDAD5AZUULI6KNCFLUKIR6GSUAULRWKOBYWEEPV4F3AFVTBYQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.984860385 (519h38m25.85594343s) [noise=true , address=145.79.227.51:4410 , piece_id=HKYBCHCG473W4AA5FCG6I7GTRHSVX4UFZ3CRSSBUZ4BIAL3ROEFQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.987339887 (519h38m25.858422932s) [noise=true , address=145.79.227.17:4972 , piece_id=SFRLUXDZQG22FQ4I3YJUGDDPEUY7D73OKS5E3ZZHLCPEXZRXJHCQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:25.990430903 (519h38m25.861513948s) [noise=true , address=145.79.227.25:4617 , piece_id=XHWXF3543XZZJ4PJBCFBRWSY25SEOB7BAH6XXAKBX3F6HS5SE2DQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.006547252 (519h38m25.877630297s) [noise=true , address=145.79.227.19:4079 , piece_id=NNJIU3TF5372X7U672NZWHUDMYA6IMQJHPLZNDB6CTX5BAQFX6AA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.012076721 (519h38m25.883159766s) [noise=true , address=145.79.227.39:4015 , piece_id=NX2RWBC6N4ZNQW54TL3U6CBTIOGFXHC2PWD5ENCBBNF2CY7T7LYA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.014708302 (519h38m25.885791347s) [noise=true , address=145.79.227.3:4149 , piece_id=UOMHON66HD2SFIPXSJTOKYYY67GZCYMDDDKFYH3BBRBELMD3RQKQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.015276093 (519h38m25.886359128s) [noise=true , address=145.79.227.16:4909 , piece_id=DKOSLYAED5I2SCRH7B3S7MEOWOSS56F2IM2U357EWWCIHQ6ERAZQ, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.268670957 (519h38m26.139753992s) [noise=true , address=45.91.116.8:4046 , piece_id=FDWTASV35EHTNDCXM27TZX5VI3I5GZ465AHQWGBEOEJVPLNSX35Q, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.308284760 (519h38m26.179367795s) [noise=true , address=77.90.31.17:4770 , piece_id=Y5KAOQER7H57ZALE57MJ4HHLMMKECYZ3A4526BENXBKXIPOBQUCA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.386027277 (519h38m26.257110312s) [noise=true , address=45.91.116.10:4082 , piece_id=4INVMH7SCMZ2EADP5NCVYSJGRA3NDNT7GVBJPC5KDNPKEQI3X5EA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.436860554 (519h38m26.307943599s) [noise=true , address=45.91.116.13:4156 , piece_id=GABYQIUFWY6AVNMAC6NA7TPQADQ36SCCBR4IPX4ASDTPQXSJUNHA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: <nil> Dec 27 07:07:26.436880010 (519h38m26.307963045s) [seg_pos={0 0}, upload=455879]: Segment reached optimal threshold of 13 pieces. Dec 27 07:07:26.437043993 (519h38m26.308127028s) [noise=true , address=77.90.31.16:4736 , piece_id=2KXPCMYZMRUOYJ54ZR6NX4UDLWEBVTXTVFGBAASWTB4WDQZG4KDA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 07:07:26.437051838 (519h38m26.308134873s) [noise=true , address=77.90.31.16:4736 , piece_id=2KXPCMYZMRUOYJ54ZR6NX4UDLWEBVTXTVFGBAASWTB4WDQZG4KDA, seg_pos={0 0}, upload=455879]: Long tail cancellation due to optimal threshold Dec 27 07:07:26.437297125 (519h38m26.30838017s) [noise=true , address=77.90.31.13:4629 , piece_id=4F5KH3Q3H7G3SJ2XNSFI56ZYF7HDT3SXFP54BWNC63Q656ASDY3A, seg_pos={0 0}, upload=455879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 07:07:26.437303206 (519h38m26.308386241s) [noise=true , address=77.90.31.13:4629 , piece_id=4F5KH3Q3H7G3SJ2XNSFI56ZYF7HDT3SXFP54BWNC63Q656ASDY3A, seg_pos={0 0}, upload=455879]: Long tail cancellation due to optimal threshold Dec 27 07:07:26.437435017 (519h38m26.308518052s) [noise=true , address=45.91.116.6:4973 , piece_id=5JTZRITUBOVJQ3I2Z6UOIS24D6O3TIJVWEYXRH4LJ6NMJNZCSJYA, seg_pos={0 0}, upload=455879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 27 07:07:26.437440528 (519h38m26.308523563s) [noise=true , address=45.91.116.6:4973 , piece_id=5JTZRITUBOVJQ3I2Z6UOIS24D6O3TIJVWEYXRH4LJ6NMJNZCSJYA, seg_pos={0 0}, upload=455879]: Long tail cancellation due to optimal threshold Dec 27 07:07:26.437474031 (519h38m26.308557066s) [seg_pos={0 0}, upload=455879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:07:26.437539256 (519h38m26.308622301s) [upload=455879]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:07:26.437543504 (519h38m26.308626549s) [upload=455879]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:07:26.724246193 (519h38m26.595329258s) [upload=455879]: Done uploading segments... Dec 27 07:07:26.724269558 (519h38m26.595352613s) [upload=455879]: Upload finished. err: <nil> Duration: 30.003838301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...................................X......X.....] duration: 58.50533977s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:07:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 58.505358086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X....] duration: 5.739046248s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:07:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 5.739091856s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........X.....X.........X.....X.................X.....X.X...] duration: 3.497185876s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:07:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.497204992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X...................................X......X....] duration: 50.654391742s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:06:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 50.654420957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X...] duration: 6.16889955s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:06:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 6.168923946s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........X.....X.........X.....X.................X.....X.X..] duration: 3.455628473s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:06:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455688887s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........X.....X.........X.....X.................X.....X.X.] duration: 3.727434896s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:06:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.727485732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X..] duration: 6.034572166s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:05:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 6.034606593s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X...................................X......X...] duration: 45.768824459s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:05:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 45.768863512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........X.....X.........X.....X.................X.....X.X] duration: 5.001540757s trace: https://jaeger.dev.storj.io/trace/6ece0b8fc6e28225 FAIL (2025-12-27 07:05:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001561107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X.] duration: 5.963214074s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:04:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 5.963240074s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X.....X.........X.....X.................X.....X.] duration: 3.645243378s trace: https://jaeger.dev.storj.io/trace/5d8ddb87b498960b FAIL (2025-12-27 07:04:40 UTC) orbiter warning: context deadline exceeded Duration: 3.645269939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [......XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.X] duration: 30.003417631s trace: https://jaeger.dev.storj.io/trace/4f9b9e445a9cc423 FAIL (2025-12-27 07:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 07:04:04.471827049 (519h35m4.342910094s) [upload=455838]: Starting upload Dec 27 07:04:04.471939194 (519h35m4.343022249s) [upload=455838]: Done starting upload Dec 27 07:04:04.472114788 (519h35m4.343197843s) [upload=455838]: Uploading segments... Dec 27 07:04:04.472239194 (519h35m4.343322239s) [upload=455838]: Got next segment. Inline: false Dec 27 07:04:04.472289811 (519h35m4.343372856s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:04:04.472296233 (519h35m4.343379278s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:04:04.718039958 (519h35m4.589122993s) [seg_pos={0 0}, upload=455838]: Begin upload segment... Dec 27 07:04:04.718097557 (519h35m4.589180592s) [seg_pos={0 0}, upload=455838]: Done begin upload segment. Dec 27 07:04:04.718142303 (519h35m4.589225348s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718233436 (519h35m4.589316481s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.718883013 (519h35m4.589966058s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719274038 (519h35m4.590357073s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.719621489 (519h35m4.590704534s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720029466 (519h35m4.591112501s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720380855 (519h35m4.59146389s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.720720142 (519h35m4.591803177s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.721108782 (519h35m4.592191827s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.722408065 (519h35m4.59349111s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.723830604 (519h35m4.594913669s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.725493020 (519h35m4.596576055s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.727169563 (519h35m4.598252608s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.728818032 (519h35m4.599901077s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.730288552 (519h35m4.601371597s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:04.731860044 (519h35m4.602943109s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Uploading piece... Dec 27 07:04:06.533259717 (519h35m6.404342763s) [upload=455838]: Next returned nil segment Dec 27 07:04:06.533281650 (519h35m6.404364685s) [upload=455838]: Waiting for error group managing segments... Dec 27 07:04:06.836935828 (519h35m6.708018873s) [noise=true , address=145.79.227.38:4979 , piece_id=G7OFOGOLMOI6SVNCGMG4BJQ6CYTVXEIXGUJZ5EVZSJV2QHWXYQEA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.844395993 (519h35m6.715479038s) [noise=true , address=145.79.227.40:4050 , piece_id=A7PF66QCYC43ACXSBZTWS663CSPWFGZTRB6CMOZU456BBROEBP7A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849237703 (519h35m6.720320748s) [noise=true , address=145.79.227.46:4601 , piece_id=QKRFFUR6K5X7WU3BY6HK67WGJ6RINLV5GTNRIXUYUT2VMPA3FAAA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849566970 (519h35m6.720650015s) [noise=true , address=145.79.227.36:4517 , piece_id=WP2R47TIMF55XZZ6PE7VPJU3GJDLZ3GAKIRBGU6UWBPVGSTQGMYA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.849899954 (519h35m6.720982989s) [noise=true , address=145.79.227.22:4510 , piece_id=XPYAY4XQDZ3UHSBF75JDHXQJGYA53QWY5EDEOTGW3ERH7SZWPB6A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.870725399 (519h35m6.741808444s) [noise=true , address=145.79.227.51:4411 , piece_id=JABXT5HMMCPS3AXAEDM3UOIJOHX7RD6DUDO4BL5V7Y53BQUY3JEQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871320883 (519h35m6.742403918s) [noise=true , address=145.79.227.35:4483 , piece_id=WCB47SA4XNS33HO73B4M6XC3FKOAW2H7FJX5C4VUXA7AFYHWQZIA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.871830263 (519h35m6.742913298s) [noise=true , address=145.79.227.49:4339 , piece_id=AQYWSSWUC6QGQC3QVGWSXDDRTFCTTHMQTULHVQLCT2AZOE6FMMBQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.890893032 (519h35m6.761976077s) [noise=true , address=145.79.227.52:4446 , piece_id=SLP32G46XBV3GW4G5QJ3UBJ4IB2G5JXIPFD4TGQL4CLYSQ4QTSQQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.908338883 (519h35m6.779421918s) [noise=true , address=77.90.31.17:4773 , piece_id=KMFHERSYTZ37U7RZTDD3BTQJAXE7FN4RKW7KM6D2X5L4PRHPXHHQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.924072513 (519h35m6.795155558s) [noise=true , address=145.79.227.28:4727 , piece_id=GC4RWUFJKTS4EWZW2XIPMMNT53EAD5PTFGNSQZARTLQGBDCTD3QQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:06.939388487 (519h35m6.810471532s) [noise=true , address=77.90.31.15:4700 , piece_id=LNL4ESZSZ443IZAWPNPN6HSDJXYQCIG46A4NW7VRQMCONEJB6ZXQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082648412 (519h35m6.953731457s) [noise=true , address=77.90.31.10:4517 , piece_id=3I43EM7S2HT2NSF3BY4NIYAPNYCRF47UXWYIQSFK2BFNSQVVPN5A, seg_pos={0 0}, upload=455838]: Done uploading piece. err: <nil> Dec 27 07:04:07.082671907 (519h35m6.953754952s) [seg_pos={0 0}, upload=455838]: Segment reached optimal threshold of 13 pieces. Dec 27 07:04:07.082909420 (519h35m6.953992465s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.082917405 (519h35m6.95400045s) [noise=true , address=77.90.31.12:4589 , piece_id=IJX7PLG455N66U7FZSGO7WM2ORTXB4UZSY74FF2Q7TT5QPRRVDGA, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083134027 (519h35m6.954217092s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083159456 (519h35m6.954242501s) [noise=true , address=145.79.227.44:4193 , piece_id=5AX7ME2OPLFVRUAORHT5MXMIOCL5KGQ5D7V3QBY4G5XOBOLQGQNQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083199572 (519h35m6.954282617s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 07:04:07.083211434 (519h35m6.954294469s) [noise=true , address=45.91.116.13:4154 , piece_id=Q7ADXOL7NZBAPMBOP3YN4XTVCSX4Y2U7LGHPTPCLXL6YA5PPQHMQ, seg_pos={0 0}, upload=455838]: Long tail cancellation due to optimal threshold Dec 27 07:04:07.083261230 (519h35m6.954344275s) [seg_pos={0 0}, upload=455838]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:04:07.083356461 (519h35m6.954439506s) [upload=455838]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:04:07.083365959 (519h35m6.954449004s) [upload=455838]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:04:07.378466910 (519h35m7.249549955s) [upload=455838]: Done uploading segments... Dec 27 07:04:07.378489233 (519h35m7.249572268s) [upload=455838]: Upload finished. err: <nil> Duration: 30.003447268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X...................................X......X..] duration: 55.603626875s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 55.603727788s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............X.....X.........X.....X.................X.....X] duration: 5.001576145s trace: https://jaeger.dev.storj.io/trace/5d8ddb87b498960b FAIL (2025-12-27 07:03:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001601104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX.] duration: 8.293333863s trace: https://jaeger.dev.storj.io/trace/36a64374731ba559 FAIL (2025-12-27 07:03:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ) context deadline exceeded ======LOGS====== 07:02:19.713260012 (519h33m19.584343057s) [upload=455816]: Starting upload Dec 27 07:02:19.713370101 (519h33m19.584453136s) [upload=455816]: Done starting upload Dec 27 07:02:19.713494889 (519h33m19.584577944s) [upload=455816]: Uploading segments... Dec 27 07:02:19.713686253 (519h33m19.584769288s) [upload=455816]: Got next segment. Inline: false Dec 27 07:02:19.713735908 (519h33m19.584818963s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:02:19.713742330 (519h33m19.584825375s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:02:19.958209063 (519h33m19.829292108s) [seg_pos={0 0}, upload=455816]: Begin upload segment... Dec 27 07:02:19.958254991 (519h33m19.829338037s) [seg_pos={0 0}, upload=455816]: Done begin upload segment. Dec 27 07:02:19.958285078 (519h33m19.829368113s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958323882 (519h33m19.829406918s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958893216 (519h33m19.829976271s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959334206 (519h33m19.830417252s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959769825 (519h33m19.830852881s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960182051 (519h33m19.831265086s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960530935 (519h33m19.83161405s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960890850 (519h33m19.831973885s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961261426 (519h33m19.832344461s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961644986 (519h33m19.832728041s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962060037 (519h33m19.833143082s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962414752 (519h33m19.833497787s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962694575 (519h33m19.83377762s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963023461 (519h33m19.834106507s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963280591 (519h33m19.834363626s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963538572 (519h33m19.834621617s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:21.584837547 (519h33m21.455920592s) [upload=455816]: Next returned nil segment Dec 27 07:02:21.584854599 (519h33m21.455937644s) [upload=455816]: Waiting for error group managing segments... Dec 27 07:02:21.960640250 (519h33m21.831723285s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.961296279 (519h33m21.832379314s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.963953889 (519h33m21.835036924s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.969000689 (519h33m21.840083724s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.973197301 (519h33m21.844280346s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.976976758 (519h33m21.848059793s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989097227 (519h33m21.860180272s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989403661 (519h33m21.860486706s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989946515 (519h33m21.86102956s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.990611040 (519h33m21.861694085s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.001852716 (519h33m21.872935761s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.246244286 (519h33m22.117327321s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266438771 (519h33m22.137521806s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266457596 (519h33m22.137540631s) [seg_pos={0 0}, upload=455816]: Segment reached optimal threshold of 13 pieces. Dec 27 07:02:22.266662517 (519h33m22.137745552s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection Dec 27 07:02:22.266668267 (519h33m22.137751313s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266863299 (519h33m22.137946344s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266901853 (519h33m22.137984898s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266906722 (519h33m22.137989757s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266913454 (519h33m22.137996489s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.267006572 (519h33m22.138089607s) [seg_pos={0 0}, upload=455816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:02:22.267090912 (519h33m22.138173957s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:02:22.267095682 (519h33m22.138178717s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:02:22.554710707 (519h33m22.425793753s) [upload=455816]: Done uploading segments... Dec 27 07:02:22.554737639 (519h33m22.425820684s) [upload=455816]: Upload finished. err: <nil> Duration: 8.293369692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X...................................X......X.] duration: 43.382065533s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:03:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 43.3820903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..............X.....X.........X.....X.................X.....] duration: 3.654062603s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:02:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.654114422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XXX] duration: 30.00278291s trace: https://jaeger.dev.storj.io/trace/36a64374731ba559 FAIL (2025-12-27 07:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ) context deadline exceeded ======LOGS====== 07:02:19.713260012 (519h33m19.584343057s) [upload=455816]: Starting upload Dec 27 07:02:19.713370101 (519h33m19.584453136s) [upload=455816]: Done starting upload Dec 27 07:02:19.713494889 (519h33m19.584577944s) [upload=455816]: Uploading segments... Dec 27 07:02:19.713686253 (519h33m19.584769288s) [upload=455816]: Got next segment. Inline: false Dec 27 07:02:19.713735908 (519h33m19.584818963s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:02:19.713742330 (519h33m19.584825375s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:02:19.958209063 (519h33m19.829292108s) [seg_pos={0 0}, upload=455816]: Begin upload segment... Dec 27 07:02:19.958254991 (519h33m19.829338037s) [seg_pos={0 0}, upload=455816]: Done begin upload segment. Dec 27 07:02:19.958285078 (519h33m19.829368113s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958323882 (519h33m19.829406918s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.958893216 (519h33m19.829976271s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959334206 (519h33m19.830417252s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.959769825 (519h33m19.830852881s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960182051 (519h33m19.831265086s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960530935 (519h33m19.83161405s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.960890850 (519h33m19.831973885s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961261426 (519h33m19.832344461s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.961644986 (519h33m19.832728041s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962060037 (519h33m19.833143082s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962414752 (519h33m19.833497787s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.962694575 (519h33m19.83377762s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963023461 (519h33m19.834106507s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963280591 (519h33m19.834363626s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:19.963538572 (519h33m19.834621617s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Uploading piece... Dec 27 07:02:21.584837547 (519h33m21.455920592s) [upload=455816]: Next returned nil segment Dec 27 07:02:21.584854599 (519h33m21.455937644s) [upload=455816]: Waiting for error group managing segments... Dec 27 07:02:21.960640250 (519h33m21.831723285s) [noise=true , address=145.79.227.48:4301 , piece_id=B6ZJ355NWJKVGMRHCDLPTVQ5TLHRIWRXWRV7X5EXDL3OXYKCLGBA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.961296279 (519h33m21.832379314s) [noise=true , address=145.79.227.15:4873 , piece_id=JNP4DFAMEUA4OL2LYBCN5UUEFGHNWAKEZC3LRKANJAJN2WL7QNPQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.963953889 (519h33m21.835036924s) [noise=true , address=145.79.227.22:4510 , piece_id=VUD7Q2FLWXUCQE5ICR4H7VOFPAS3FGRM5QMTYBQ65YLUHMMDJSQQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.969000689 (519h33m21.840083724s) [noise=true , address=145.79.227.2:4113 , piece_id=2PM53AU2UMCNF72P3XEZ67ZWONLLHIBDJS4UMTIEBIJQ5CDMLIVA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.973197301 (519h33m21.844280346s) [noise=true , address=145.79.227.52:4445 , piece_id=BQPXX7AEDTU6DJQ6BQMXGJJMA5CAWPVTL2R2DXN7DBHYHH4FLTGA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.976976758 (519h33m21.848059793s) [noise=true , address=145.79.227.47:4265 , piece_id=U2FGVZXAH6E4P6S6XJAAO36DVWRTV4KHRWEA7XNSAHP7BFRORCXA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989097227 (519h33m21.860180272s) [noise=true , address=145.79.227.39:4015 , piece_id=SQKJKVAC67PCCHWG7TUVL5WFY2ESBCFBO54RRNKFEERC7A2YEYHA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989403661 (519h33m21.860486706s) [noise=true , address=145.79.227.38:4978 , piece_id=2SLWW4FMNQ3VB7RF2JXJJ6DML5OW42MEU32HF5ZQ2VN3UOE5Z3UA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.989946515 (519h33m21.86102956s) [noise=true , address=145.79.227.46:4603 , piece_id=FFOCZZGSI5JKRD64ACARYPZLQWLVCBYS7U7NHIRCPUQXRWTRGD7Q, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:21.990611040 (519h33m21.861694085s) [noise=true , address=145.79.227.51:4409 , piece_id=PCAPISRSIIKBUSXJ7WBVHNXCYSDFFGGKXX2Z3AL75HO2NVDBKUCA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.001852716 (519h33m21.872935761s) [noise=true , address=145.79.227.49:4339 , piece_id=G56JGR6G3B3ZE4O77NLLQAIBQCCDTHBP2MJ4PNGHLBRI2XSYNHZQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.246244286 (519h33m22.117327321s) [noise=true , address=77.90.31.15:4700 , piece_id=7SCCLCUTMVT66YAGTD674TPILTFQVDPJ3CPEIQ6AORE3Y5SIAJHQ, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266438771 (519h33m22.137521806s) [noise=true , address=45.91.116.9:4264 , piece_id=EWPDB5JOC5QVJMC2H4DS66FHCKRXGPRVM2WFOFHAP2FIGMAQC4GA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: <nil> Dec 27 07:02:22.266457596 (519h33m22.137540631s) [seg_pos={0 0}, upload=455816]: Segment reached optimal threshold of 13 pieces. Dec 27 07:02:22.266662517 (519h33m22.137745552s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47064->45.91.116.5:4298: use of closed network connection Dec 27 07:02:22.266668267 (519h33m22.137751313s) [noise=true , address=45.91.116.5:4298 , piece_id=67PVEPHD26JSTCQVZGVBX4NVL7PWXUNGXFTS2W3XLEHKVFPUTZ4A, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266863299 (519h33m22.137946344s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266901853 (519h33m22.137984898s) [noise=true , address=45.91.116.8:4047 , piece_id=XOLPJQCEUUSFQ2MFNX7SSJZ5AT4U6SZAJK4JS74ID4Y2KHZ3MVIA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.266906722 (519h33m22.137989757s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 27 07:02:22.266913454 (519h33m22.137996489s) [noise=true , address=145.79.227.44:4195 , piece_id=NOOCNQJPVXQEJ236QLWLZRWEOGS3ETVOFIGUA47QWIFKBDJDOVZA, seg_pos={0 0}, upload=455816]: Long tail cancellation due to optimal threshold Dec 27 07:02:22.267006572 (519h33m22.138089607s) [seg_pos={0 0}, upload=455816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:02:22.267090912 (519h33m22.138173957s) [upload=455816]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:02:22.267095682 (519h33m22.138178717s) [upload=455816]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:02:22.554710707 (519h33m22.425793753s) [upload=455816]: Done uploading segments... Dec 27 07:02:22.554737639 (519h33m22.425820684s) [upload=455816]: Upload finished. err: <nil> Duration: 30.002808429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X...................................X......X] duration: 1m0.003418863s trace: https://jaeger.dev.storj.io/trace/f85fd2d95537437 FAIL (2025-12-27 07:02:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.962742940 (519h32m21.833825985s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963104960 (519h32m21.834187995s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963443263 (519h32m21.834526318s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963492187 (519h32m21.834575222s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.963877400 (519h32m21.834960536s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964341263 (519h32m21.835424308s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964402370 (519h32m21.835485405s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964693965 (519h32m21.835777s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.964981223 (519h32m21.836064258s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965015247 (519h32m21.836098293s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965378378 (519h32m21.836461414s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965747361 (519h32m21.836830397s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.965875957 (519h32m21.836959002s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966369546 (519h32m21.837452591s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:21.966559688 (519h32m21.837642723s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Uploading piece... Dec 27 07:01:23.422417540 (519h32m23.293500586s) [upload=455804]: Next returned nil segment Dec 27 07:01:23.422434272 (519h32m23.293517307s) [upload=455804]: Waiting for error group managing segments... Dec 27 07:01:23.857113116 (519h32m23.728196162s) [noise=true , address=145.79.227.8:4347 , piece_id=2KTGKJD5PUCBEBBOX4KSD2JXPN3ADZGFQXBHDUX5KEAVTELGGY7A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:23.980050734 (519h32m23.851133779s) [noise=true , address=145.79.227.49:4361 , piece_id=YJ6A7UPMAMTUKUJAJE4APYT4ES7Z3JILCXD66HMNLKW7LWWWXIJQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.010585368 (519h32m23.881668403s) [noise=true , address=77.90.31.15:4702 , piece_id=XXMKU2J62SFYTAPAAVY27SYRQMMTTZOIBNY2E62VYTJV66WOEVAA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.142475193 (519h32m24.013558238s) [noise=true , address=77.90.31.19:4869 , piece_id=5G6J7EVRGZMZPMI5IBH4GZ6PE5DGAWE6YY6MEQECZCNRK3TSZITA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.160675280 (519h32m24.031758325s) [noise=true , address=77.90.31.14:4679 , piece_id=63U4JKP44KOY6I3V7EO23B5EZ3PCXD2VJ6AYP7DZ5QGIHR2UGZEA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.162062672 (519h32m24.033145717s) [noise=true , address=45.91.116.14:4221 , piece_id=7CQFOYC57TQCBK4DI7QOGMWXCOMLNH7FA5SYGA2AID4XUO7ALEGQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.205510034 (519h32m24.076593069s) [noise=true , address=77.90.31.13:4638 , piece_id=QXVCHD47MR2CE337QVAD7W6FTCC52RUCACV2DBY63JP7T6Y7GNCQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.329146903 (519h32m24.200229938s) [noise=true , address=77.90.31.16:4737 , piece_id=7SST7QGJ24UUBO6UYKIHRDTCZ3GCYLVM3OM2QWJSEXUH2D6SI2HQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.493081757 (519h32m24.364164792s) [noise=true , address=45.91.116.13:4178 , piece_id=XWZRJRYZKRSQV6D4IJBU7ZL5OIE6XSWFHR4V5637C7WB2D3X7HDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.760352444 (519h32m24.631435479s) [noise=true , address=145.79.227.30:4790 , piece_id=PJ3BEXN6ZJOM2WNUTA3J63CQJIMADWRTBC5FSWQB72FS2Q47BGDA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:24.796726858 (519h32m24.667809903s) [noise=true , address=145.79.227.44:4212 , piece_id=4NTFJVD7E2PCOID4EPIO2DDY2UP5NC676SBWEB3KULMI65NTV7XA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.047568762 (519h32m24.918651797s) [noise=true , address=77.90.31.12:4621 , piece_id=A242NNNAOP23ZRZJ57G7D7BGHP7J6U3GOWA3C5UWAGZ6JX3A3ULA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.048262453 (519h32m24.919345488s) [noise=true , address=77.90.31.18:4818 , piece_id=KEIV2YCI4LECB6DB4RGDCFVLYVXZA5A4YYI5KF2NO56LQOIPFLTQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082534503 (519h32m24.953617548s) [noise=true , address=193.149.164.143:4697 , piece_id=TP2I7AC56N6LFJXKU6CVMVVBCLPRHLCA27VTNIPPHETBJAKEEU7Q, seg_pos={0 0}, upload=455804]: Done uploading piece. err: <nil> Dec 27 07:01:25.082559760 (519h32m24.953642805s) [seg_pos={0 0}, upload=455804]: Segment reached optimal threshold of 14 pieces. Dec 27 07:01:25.082903896 (519h32m24.953986951s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.082915278 (519h32m24.953998313s) [noise=true , address=145.79.227.40:4073 , piece_id=A3CYPV75WV3PCXI6LGV6YRVTOQUNSS5MQUILOIWSXPQ5ZHRZ26ZA, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083102775 (519h32m24.95418582s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16KXs6ZXZyroRJibAYU5bsga1RFSpmf7yEtaNG7tamaX7QKvng): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083121501 (519h32m24.954204546s) [noise=true , address=45.91.116.6:4008 , piece_id=OGMU6MWNHQFQEEQAEGAN4ELJIAQYIZHA6I7HXBLHIC76PQAINI3A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083161076 (519h32m24.954244121s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:25.083171766 (519h32m24.954254811s) [noise=true , address=77.90.31.21:4894 , piece_id=GIPXPZBDKZ3QTTBCZ3DQHNUIQJ4EAOVXCBATOPZLWBURLV4KN76A, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083298137 (519h32m24.954381172s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33956->64.49.13.169:4715: use of closed network connection Dec 27 07:01:25.083307014 (519h32m24.954390049s) [noise=true , address=64.49.13.169:4715 , piece_id=R7EEYN4PDAP4FWTJMR7T3YYV6FZNB4UAEHMISP4ARFJW4WZCZXAQ, seg_pos={0 0}, upload=455804]: Long tail cancellation due to optimal threshold Dec 27 07:01:25.083339866 (519h32m24.954422911s) [seg_pos={0 0}, upload=455804]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 07:01:25.083443474 (519h32m24.954526519s) [upload=455804]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:25.083449816 (519h32m24.954532851s) [upload=455804]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:25.380069709 (519h32m25.251152754s) [upload=455804]: Done uploading segments... Dec 27 07:01:25.380092553 (519h32m25.251175598s) [upload=455804]: Upload finished. err: <nil> Duration: 1m0.003436557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...............X.....X.........X.....X.................X....] duration: 3.734486175s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:02:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.734538895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..XX] duration: 30.004704506s trace: https://jaeger.dev.storj.io/trace/287bb601b51eaac FAIL (2025-12-27 07:01:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 07:01:19.259902587 (519h32m19.130985632s) [upload=455803]: Starting upload Dec 27 07:01:19.259958274 (519h32m19.131041309s) [upload=455803]: Done starting upload Dec 27 07:01:19.260173103 (519h32m19.131256148s) [upload=455803]: Uploading segments... Dec 27 07:01:19.260257703 (519h32m19.131340738s) [upload=455803]: Got next segment. Inline: false Dec 27 07:01:19.260288272 (519h32m19.131371307s) [upload=455803]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:01:19.260292349 (519h32m19.131375384s) [upload=455803]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:01:19.513066440 (519h32m19.384149485s) [seg_pos={0 0}, upload=455803]: Begin upload segment... Dec 27 07:01:19.513120362 (519h32m19.384203397s) [seg_pos={0 0}, upload=455803]: Done begin upload segment. Dec 27 07:01:19.513136554 (519h32m19.384219589s) [noise=true , address=145.79.227.47:4265 , piece_id=LG44OFTAQU2GWGFTIEWFRS5HKYD3GFZ6USTKT4B4HPRQXDXL64PA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.513194684 (519h32m19.384277719s) [noise=true , address=145.79.227.35:4482 , piece_id=Z3KC5QOZ3H25DPWWNCRCTOKY46GPN4KLEFBMS4XLMM7PE2N6HQ7Q, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.513950603 (519h32m19.385033649s) [noise=true , address=77.90.31.18:4808 , piece_id=DTO6GBRYT2OFLLMGAWBY5EUJ2SFDATNYU2FPJRVPDMVD334BGC6A, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.514250074 (519h32m19.385333109s) [noise=true , address=145.79.227.29:4943 , piece_id=O7GQ4GSRF7VD4CJWCQXB4BTYHPDT77YYSNLYPFRGP2DKPJVNTRUA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.514573660 (519h32m19.385656695s) [noise=true , address=145.79.227.5:4221 , piece_id=5ERYTHGIAVX6G2RQXHGN2NARS47E4VYATEOCUXQLLPOKB6EK3Q6A, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.514852912 (519h32m19.385935947s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRZNVGWCTS33W45BCKB2E2U55HLP6RAERARMLG7OBRXYJPTYBWQ, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.515123427 (519h32m19.386206462s) [noise=true , address=145.79.227.51:4409 , piece_id=NPW7PTV3OUQCSFVU6OVWBGZQNM3TBZN7NKEO5G46QH55O4JEKALA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.515393031 (519h32m19.386476066s) [noise=true , address=145.79.227.38:4977 , piece_id=QNDAQVLKEXT2KOZOJS6UHV3DBFYQ5LSFR2HJJ4PPWPHTTQDM2V4Q, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.515910987 (519h32m19.386994032s) [noise=true , address=77.90.31.11:4554 , piece_id=5DW2GCZCBHLV2B2V24NGXI7P5RESP3ELE2OKCMG6JCITCLHUWWEA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.516846598 (519h32m19.387929643s) [noise=true , address=145.79.227.28:4727 , piece_id=LAJ5TPN33HCGQWEGUGCEQXN435HLE36OT4QIU5YKTX4TGCIVKJQQ, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.517128805 (519h32m19.388211841s) [noise=true , address=145.79.227.36:4519 , piece_id=ABCM3AWPVC3PGQSZYIP2VZRV7SCU7NXKZ5IH6MSTARENCVIMPBRA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.518056111 (519h32m19.389139146s) [noise=true , address=145.79.227.12:4765 , piece_id=33PMSKVSDLMZXOHMQ6IIO62YJRH7JKHCD44252NQLGXYEG7QGOMA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.518393394 (519h32m19.389476429s) [noise=true , address=45.91.116.11:4118 , piece_id=W52N7H4U3YUSTINBSG5FXHT7ZKJ2X2W77FEOB4H7UR5DWAE3VOKA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.519148031 (519h32m19.390231066s) [noise=true , address=77.90.31.17:4770 , piece_id=YTUPGZMZUZNU66QLUO7CMSO3HCDXWV5HDICX4UX7LMBVJWTV4Y3A, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.520245640 (519h32m19.391328676s) [noise=true , address=145.79.227.52:4445 , piece_id=JLILBLAGC36AHPA43EQL3CXTERHSYSOCN7MJ2LC5U2DFHOBZ72SA, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:19.520710816 (519h32m19.391793861s) [noise=true , address=145.79.227.6:4259 , piece_id=RLDWE26JG6IJZ6OUBJOD2WAW7GKOHCOTT4334L5RP3WGCLZXMV5Q, seg_pos={0 0}, upload=455803]: Uploading piece... Dec 27 07:01:21.141571877 (519h32m21.012654912s) [upload=455803]: Next returned nil segment Dec 27 07:01:21.141589611 (519h32m21.012672656s) [upload=455803]: Waiting for error group managing segments... Dec 27 07:01:21.525850640 (519h32m21.396933675s) [noise=true , address=145.79.227.36:4519 , piece_id=ABCM3AWPVC3PGQSZYIP2VZRV7SCU7NXKZ5IH6MSTARENCVIMPBRA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.528154137 (519h32m21.399237172s) [noise=true , address=145.79.227.35:4482 , piece_id=Z3KC5QOZ3H25DPWWNCRCTOKY46GPN4KLEFBMS4XLMM7PE2N6HQ7Q, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.528640874 (519h32m21.399723909s) [noise=true , address=145.79.227.38:4977 , piece_id=QNDAQVLKEXT2KOZOJS6UHV3DBFYQ5LSFR2HJJ4PPWPHTTQDM2V4Q, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.529180741 (519h32m21.400263776s) [noise=true , address=145.79.227.5:4221 , piece_id=5ERYTHGIAVX6G2RQXHGN2NARS47E4VYATEOCUXQLLPOKB6EK3Q6A, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.530809713 (519h32m21.401892759s) [noise=true , address=145.79.227.12:4765 , piece_id=33PMSKVSDLMZXOHMQ6IIO62YJRH7JKHCD44252NQLGXYEG7QGOMA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.531873480 (519h32m21.402956525s) [noise=true , address=145.79.227.51:4409 , piece_id=NPW7PTV3OUQCSFVU6OVWBGZQNM3TBZN7NKEO5G46QH55O4JEKALA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.541565735 (519h32m21.41264879s) [noise=true , address=145.79.227.52:4445 , piece_id=JLILBLAGC36AHPA43EQL3CXTERHSYSOCN7MJ2LC5U2DFHOBZ72SA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.543842721 (519h32m21.414925756s) [noise=true , address=145.79.227.28:4727 , piece_id=LAJ5TPN33HCGQWEGUGCEQXN435HLE36OT4QIU5YKTX4TGCIVKJQQ, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.548417933 (519h32m21.419500978s) [noise=true , address=145.79.227.29:4943 , piece_id=O7GQ4GSRF7VD4CJWCQXB4BTYHPDT77YYSNLYPFRGP2DKPJVNTRUA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.551008146 (519h32m21.422091191s) [noise=true , address=145.79.227.6:4259 , piece_id=RLDWE26JG6IJZ6OUBJOD2WAW7GKOHCOTT4334L5RP3WGCLZXMV5Q, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.553580584 (519h32m21.424663619s) [noise=true , address=145.79.227.47:4265 , piece_id=LG44OFTAQU2GWGFTIEWFRS5HKYD3GFZ6USTKT4B4HPRQXDXL64PA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.706087037 (519h32m21.577170082s) [noise=true , address=77.90.31.18:4808 , piece_id=DTO6GBRYT2OFLLMGAWBY5EUJ2SFDATNYU2FPJRVPDMVD334BGC6A, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.985193130 (519h32m21.856276195s) [noise=true , address=45.91.116.10:4081 , piece_id=HWRZNVGWCTS33W45BCKB2E2U55HLP6RAERARMLG7OBRXYJPTYBWQ, seg_pos={0 0}, upload=455803]: Done uploading piece. err: <nil> Dec 27 07:01:21.985216946 (519h32m21.856299991s) [seg_pos={0 0}, upload=455803]: Segment reached optimal threshold of 13 pieces. Dec 27 07:01:21.985542405 (519h32m21.85662544s) [noise=true , address=45.91.116.11:4118 , piece_id=W52N7H4U3YUSTINBSG5FXHT7ZKJ2X2W77FEOB4H7UR5DWAE3VOKA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:21.985549979 (519h32m21.856633024s) [noise=true , address=45.91.116.11:4118 , piece_id=W52N7H4U3YUSTINBSG5FXHT7ZKJ2X2W77FEOB4H7UR5DWAE3VOKA, seg_pos={0 0}, upload=455803]: Long tail cancellation due to optimal threshold Dec 27 07:01:21.985782893 (519h32m21.856865928s) [noise=true , address=77.90.31.17:4770 , piece_id=YTUPGZMZUZNU66QLUO7CMSO3HCDXWV5HDICX4UX7LMBVJWTV4Y3A, seg_pos={0 0}, upload=455803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 07:01:21.985836675 (519h32m21.85691972s) [noise=true , address=77.90.31.17:4770 , piece_id=YTUPGZMZUZNU66QLUO7CMSO3HCDXWV5HDICX4UX7LMBVJWTV4Y3A, seg_pos={0 0}, upload=455803]: Long tail cancellation due to optimal threshold Dec 27 07:01:21.985938649 (519h32m21.857021684s) [noise=true , address=77.90.31.11:4554 , piece_id=5DW2GCZCBHLV2B2V24NGXI7P5RESP3ELE2OKCMG6JCITCLHUWWEA, seg_pos={0 0}, upload=455803]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:59798->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:59798->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59798->77.90.31.11:4554: use of closed network connection Dec 27 07:01:21.985944221 (519h32m21.857027266s) [noise=true , address=77.90.31.11:4554 , piece_id=5DW2GCZCBHLV2B2V24NGXI7P5RESP3ELE2OKCMG6JCITCLHUWWEA, seg_pos={0 0}, upload=455803]: Long tail cancellation due to optimal threshold Dec 27 07:01:21.985959680 (519h32m21.857042725s) [seg_pos={0 0}, upload=455803]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:01:21.986019263 (519h32m21.857102298s) [upload=455803]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:01:21.986023702 (519h32m21.857106747s) [upload=455803]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:01:22.287013235 (519h32m22.15809628s) [upload=455803]: Done uploading segments... Dec 27 07:01:22.287034145 (519h32m22.15811719s) [upload=455803]: Upload finished. err: <nil> Duration: 30.004717641s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [................X.....X.........X.....X.................X...] duration: 3.48726263s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:01:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487295613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..X] duration: 30.002373838s trace: https://jaeger.dev.storj.io/trace/316840e12da9a815 FAIL (2025-12-27 07:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: VVG4WFOKMXX2FMY276L3I4ZW3GYJJBNHVDXIFHHYL4RRE442V2PA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ZFKUB3ZM6J6EEF4NDOIKZFWZVIESE5SUJVWX4RNLOBQXLLLUQSQQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: XJP5TGA57E3SXAMP6UB577YJN76XN2WLVUOELICWUX5QXP2VJNZQ) context deadline exceeded ======LOGS====== 27 07:00:49.051038801 (519h31m48.922121846s) [upload=455797]: Starting upload Dec 27 07:00:49.051093896 (519h31m48.922176941s) [upload=455797]: Done starting upload Dec 27 07:00:49.051159561 (519h31m48.922242606s) [upload=455797]: Uploading segments... Dec 27 07:00:49.051273167 (519h31m48.922356203s) [upload=455797]: Got next segment. Inline: false Dec 27 07:00:49.051325958 (519h31m48.922408994s) [upload=455797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 07:00:49.051330557 (519h31m48.922413602s) [upload=455797]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 07:00:49.320639445 (519h31m49.19172248s) [seg_pos={0 0}, upload=455797]: Begin upload segment... Dec 27 07:00:49.320690141 (519h31m49.191773176s) [seg_pos={0 0}, upload=455797]: Done begin upload segment. Dec 27 07:00:49.320733494 (519h31m49.191816539s) [noise=true , address=45.91.116.6:4973 , piece_id=XJP5TGA57E3SXAMP6UB577YJN76XN2WLVUOELICWUX5QXP2VJNZQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.320757410 (519h31m49.191840455s) [noise=true , address=145.79.227.37:4565 , piece_id=24P75DDZUFUV7WK4B6Y42JO46MD5SEAAGQKQGDXG2QAYU4FSRRVA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.320801553 (519h31m49.191884598s) [noise=true , address=145.79.227.47:4265 , piece_id=AWO56KIT6PXOYL2YS3A4VDNWN4NNPDYB6A2TW4FU7CKH7ZLSZXZQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.321389032 (519h31m49.192472067s) [noise=true , address=77.90.31.17:4773 , piece_id=OCLGYW6UVKBIJJ2YYBQBUXOF3K3D44RRSKXXONRLDI37JN2QR6SQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.321782281 (519h31m49.192865326s) [noise=true , address=45.91.116.8:4048 , piece_id=HPMAPUDETNCNPEU3HHWW5SM4S43T7JNLTMXFVKY4KJGGPPBDMDKA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.322306610 (519h31m49.193389655s) [noise=true , address=145.79.227.11:4730 , piece_id=KWQSIAXDKDGDG2IUVZRIVXUQE6BIY2ELST6TDI7234PO5MSEUDAA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.322747489 (519h31m49.193830534s) [noise=true , address=145.79.227.15:4873 , piece_id=AUZGGAQLDFEDFOQ5UEUMNCHU6JIJ6IUNBITWTZQHIIKKQNHZ2EHQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.323325420 (519h31m49.194408455s) [noise=true , address=145.79.227.22:4510 , piece_id=DB3YZIBIB5E7DYRRGHHVQLPQ56AWUXGL343CJR5ZMHRWTQVRZUHQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.323876650 (519h31m49.194959695s) [noise=true , address=45.91.116.11:4118 , piece_id=RJRLHU642TIMM772PI43E32LYFZ46QVKHGFBGCAURCDYGJ5V35RQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.324536536 (519h31m49.195619581s) [noise=true , address=77.90.31.16:4736 , piece_id=VD7VF7SLPADS5DSV74B5H6YS352ZYCGVYHP4X47C6GTJG6ZWMCNQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.325787578 (519h31m49.196870623s) [noise=true , address=77.90.31.14:4664 , piece_id=VVG4WFOKMXX2FMY276L3I4ZW3GYJJBNHVDXIFHHYL4RRE442V2PA, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.327132930 (519h31m49.198215966s) [noise=true , address=145.79.227.5:4221 , piece_id=74T4VJXCRIL4FZ4ALWTT4YHPMXXBWKU3IMYZ7IOF2ZGPIAKVE7TQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.329103753 (519h31m49.200186798s) [noise=true , address=145.79.227.6:4259 , piece_id=GB42TKIDAU6EWCANCTOAVZN7QI2CIGQEBCWQNNMJX3MHACRJNH2A, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.331084785 (519h31m49.20216782s) [noise=true , address=145.79.227.36:4518 , piece_id=R3KAQG6KRMBOBBA5HXNL57JY43QQC7UYGYNW66J7IZMQE7VYLS3A, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.332720039 (519h31m49.203803094s) [noise=true , address=77.90.31.11:4555 , piece_id=JPPQSNBMC336XVRHF3AIGDVI7SYDRFFNFJ5RWQZELJZ5Y4WXWT6A, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:49.337706164 (519h31m49.208789209s) [noise=true , address=45.91.116.9:4264 , piece_id=ZFKUB3ZM6J6EEF4NDOIKZFWZVIESE5SUJVWX4RNLOBQXLLLUQSQQ, seg_pos={0 0}, upload=455797]: Uploading piece... Dec 27 07:00:51.049779524 (519h31m50.920862579s) [upload=455797]: Next returned nil segment Dec 27 07:00:51.049834658 (519h31m50.920917703s) [upload=455797]: Waiting for error group managing segments... Dec 27 07:00:51.336427749 (519h31m51.207510794s) [noise=true , address=145.79.227.15:4873 , piece_id=AUZGGAQLDFEDFOQ5UEUMNCHU6JIJ6IUNBITWTZQHIIKKQNHZ2EHQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.337125528 (519h31m51.208208573s) [noise=true , address=145.79.227.37:4565 , piece_id=24P75DDZUFUV7WK4B6Y42JO46MD5SEAAGQKQGDXG2QAYU4FSRRVA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.345051581 (519h31m51.216134626s) [noise=true , address=145.79.227.5:4221 , piece_id=74T4VJXCRIL4FZ4ALWTT4YHPMXXBWKU3IMYZ7IOF2ZGPIAKVE7TQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.348034280 (519h31m51.219117316s) [noise=true , address=145.79.227.47:4265 , piece_id=AWO56KIT6PXOYL2YS3A4VDNWN4NNPDYB6A2TW4FU7CKH7ZLSZXZQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.348764771 (519h31m51.219847847s) [noise=true , address=145.79.227.11:4730 , piece_id=KWQSIAXDKDGDG2IUVZRIVXUQE6BIY2ELST6TDI7234PO5MSEUDAA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.350353636 (519h31m51.221436671s) [noise=true , address=145.79.227.22:4510 , piece_id=DB3YZIBIB5E7DYRRGHHVQLPQ56AWUXGL343CJR5ZMHRWTQVRZUHQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.359355668 (519h31m51.230438713s) [noise=true , address=145.79.227.6:4259 , piece_id=GB42TKIDAU6EWCANCTOAVZN7QI2CIGQEBCWQNNMJX3MHACRJNH2A, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.365285059 (519h31m51.236368094s) [noise=true , address=145.79.227.36:4518 , piece_id=R3KAQG6KRMBOBBA5HXNL57JY43QQC7UYGYNW66J7IZMQE7VYLS3A, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.564273793 (519h31m51.435356828s) [noise=true , address=45.91.116.9:4264 , piece_id=ZFKUB3ZM6J6EEF4NDOIKZFWZVIESE5SUJVWX4RNLOBQXLLLUQSQQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.632836340 (519h31m51.503919385s) [noise=true , address=77.90.31.14:4664 , piece_id=VVG4WFOKMXX2FMY276L3I4ZW3GYJJBNHVDXIFHHYL4RRE442V2PA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.658615609 (519h31m51.529698654s) [noise=true , address=45.91.116.6:4973 , piece_id=XJP5TGA57E3SXAMP6UB577YJN76XN2WLVUOELICWUX5QXP2VJNZQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.737596855 (519h31m51.6086799s) [noise=true , address=45.91.116.11:4118 , piece_id=RJRLHU642TIMM772PI43E32LYFZ46QVKHGFBGCAURCDYGJ5V35RQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.960622530 (519h31m51.831705565s) [noise=true , address=77.90.31.11:4555 , piece_id=JPPQSNBMC336XVRHF3AIGDVI7SYDRFFNFJ5RWQZELJZ5Y4WXWT6A, seg_pos={0 0}, upload=455797]: Done uploading piece. err: <nil> Dec 27 07:00:51.960644462 (519h31m51.831727497s) [seg_pos={0 0}, upload=455797]: Segment reached optimal threshold of 13 pieces. Dec 27 07:00:51.960857908 (519h31m51.831940953s) [noise=true , address=77.90.31.16:4736 , piece_id=VD7VF7SLPADS5DSV74B5H6YS352ZYCGVYHP4X47C6GTJG6ZWMCNQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 27 07:00:51.960868107 (519h31m51.831951162s) [noise=true , address=77.90.31.16:4736 , piece_id=VD7VF7SLPADS5DSV74B5H6YS352ZYCGVYHP4X47C6GTJG6ZWMCNQ, seg_pos={0 0}, upload=455797]: Long tail cancellation due to optimal threshold Dec 27 07:00:51.960973899 (519h31m51.832056934s) [noise=true , address=77.90.31.17:4773 , piece_id=OCLGYW6UVKBIJJ2YYBQBUXOF3K3D44RRSKXXONRLDI37JN2QR6SQ, seg_pos={0 0}, upload=455797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:56524->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:56524->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56524->77.90.31.17:4773: use of closed network connection Dec 27 07:00:51.960990891 (519h31m51.832073936s) [noise=true , address=77.90.31.17:4773 , piece_id=OCLGYW6UVKBIJJ2YYBQBUXOF3K3D44RRSKXXONRLDI37JN2QR6SQ, seg_pos={0 0}, upload=455797]: Long tail cancellation due to optimal threshold Dec 27 07:00:51.961152900 (519h31m51.832235945s) [noise=true , address=45.91.116.8:4048 , piece_id=HPMAPUDETNCNPEU3HHWW5SM4S43T7JNLTMXFVKY4KJGGPPBDMDKA, seg_pos={0 0}, upload=455797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 07:00:51.961162178 (519h31m51.832245213s) [noise=true , address=45.91.116.8:4048 , piece_id=HPMAPUDETNCNPEU3HHWW5SM4S43T7JNLTMXFVKY4KJGGPPBDMDKA, seg_pos={0 0}, upload=455797]: Long tail cancellation due to optimal threshold Dec 27 07:00:51.961180352 (519h31m51.832263397s) [seg_pos={0 0}, upload=455797]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 07:00:51.961249443 (519h31m51.832332488s) [upload=455797]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 07:00:51.961254163 (519h31m51.832337208s) [upload=455797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 07:00:52.259259713 (519h31m52.130342758s) [upload=455797]: Done uploading segments... Dec 27 07:00:52.259282738 (519h31m52.130365783s) [upload=455797]: Upload finished. err: <nil> Duration: 30.002394547s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X...................................X......] duration: 54.406803529s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 07:01:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 54.406820661s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.................X.....X.........X.....X.................X..] duration: 3.522128462s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:01:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522149322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X.....X.........X.....X.................X.] duration: 3.35444426s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:00:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.354469858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...................................X.....] duration: 50.892026398s trace: https://jaeger.dev.storj.io/trace/5da5d4c386092dd FAIL (2025-12-27 07:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2, Piece ID: RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ) context deadline exceeded; piecestore: (Node ID: 1afoGj5ZD9twztzCN957MdgkWwydWT3pkGkXXtStw3P6hZBoqK, Piece ID: 4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q) context deadline exceeded; piecestore: (Node ID: 1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, Piece ID: 2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ) context deadline exceeded ======LOGS====== ise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.539569838 (519h25m33.410652873s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.540158970 (519h25m33.411242005s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541272471 (519h25m33.412355516s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.541754949 (519h25m33.412837995s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.542851707 (519h25m33.413934742s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.543193979 (519h25m33.414277024s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544204664 (519h25m33.415287719s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.544517710 (519h25m33.415600755s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.545774363 (519h25m33.416857398s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.546080516 (519h25m33.417163571s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.547616140 (519h25m33.418699225s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.548806658 (519h25m33.419889703s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.551907662 (519h25m33.422990697s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.554937872 (519h25m33.426020917s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:33.561287874 (519h25m33.432370909s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Uploading piece... Dec 27 06:54:35.005190704 (519h25m34.876273759s) [upload=455715]: Next returned nil segment Dec 27 06:54:35.005203388 (519h25m34.876286433s) [upload=455715]: Waiting for error group managing segments... Dec 27 06:54:35.395370023 (519h25m35.266453058s) [noise=true , address=145.79.227.16:4923 , piece_id=5VXFBR5BJADXFPF5E4R6XZT426SXEXWXNWUXO4JHIMET6NI4W4LA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.443781127 (519h25m35.314864163s) [noise=true , address=145.79.227.33:4897 , piece_id=M5VA5LZJ7N6XLOAEXBHAAFK53R7UWESDHYMDBVKU7VE5AXFDA7FQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.458745212 (519h25m35.329828257s) [noise=true , address=145.79.227.25:4645 , piece_id=YNWRIZSMFZFXZQTFJGFMINFHICXDV3XBEGFOHIJNNWDGB5IVKJBQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.616491773 (519h25m35.487574808s) [noise=true , address=77.90.31.19:4853 , piece_id=QTKBEUNBKUNLMGXFQBX7TOCDGFWZ7IGDQ4CYI7M33C74JD6YFLJA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.682853990 (519h25m35.553937025s) [noise=true , address=77.90.31.15:4712 , piece_id=2UTFZXWROYI3DQIRS4RD55J3ZP6V72TIBPSJLEFWMJI3YS6IJ7ZQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.687928242 (519h25m35.559011277s) [noise=true , address=77.90.31.16:4747 , piece_id=RPTPOZ2T6QWICSEIW7YIX3U6HXRJWMTEKUEFB4TDICLQADSS5GEQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.695492126 (519h25m35.566575171s) [noise=true , address=77.90.31.14:4677 , piece_id=4KQGIXJIZUKO6VQP5QW2A62RYEG5NDT74UFODRXBTKOSA3AEDP5Q, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:35.730035864 (519h25m35.601118899s) [noise=true , address=77.90.31.12:4597 , piece_id=FLO4D54R22IQPETPDUKIHNF5D4EN6VK7A5CYBTTEN7V3OW4E27VQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.013933181 (519h25m35.885016216s) [noise=true , address=145.79.227.15:4880 , piece_id=X3CNBVFSSYSZDJAS2CRFMKG3ZRPJK4II4FQ4WC3TA4QYA3SFAV4A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.298455339 (519h25m36.169538374s) [noise=true , address=145.79.227.44:4208 , piece_id=EPQWWISRYM77BU2FF3VFDF43KV256ABTHNKKDODXTZ5F7RYLOJCQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.314682147 (519h25m36.185765192s) [noise=true , address=145.79.227.43:4165 , piece_id=HLA7JKTPOG64WL6HWSSTGK4TFM5CH524VT4WP35AUO632EP77YAQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.328825780 (519h25m36.199908815s) [noise=true , address=145.79.227.30:4781 , piece_id=THYS5DVR4S7OI2UXWTEWV75CFPYAABMXYJ7AQLUFWRKXJYZNWPBA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334786269 (519h25m36.205869314s) [noise=true , address=145.79.227.40:4081 , piece_id=4O5IZF26QK6VBCAZ7GDDYBSRBT6F2LBSPFRVLZX5I5KGDIX26WGQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334873185 (519h25m36.20595622s) [noise=true , address=145.79.227.43:4184 , piece_id=52TQL6KU73VPSMY6CVO7OG6LDCHFB5DJ2OICBIJGIWXS7MUBSMFQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: <nil> Dec 27 06:54:36.334890297 (519h25m36.205973332s) [seg_pos={0 0}, upload=455715]: Segment reached optimal threshold of 14 pieces. Dec 27 06:54:36.335045914 (519h25m36.206128949s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40992->64.49.13.169:4706: use of closed network connection Dec 27 06:54:36.335053999 (519h25m36.206137035s) [noise=true , address=64.49.13.169:4706 , piece_id=PULHGULNRYFPJJHUTOEMYYFELFKHBYQUBOQUGUIPTQ7DUEJDYOMQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335084828 (519h25m36.206167863s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335094306 (519h25m36.206177341s) [noise=true , address=77.90.31.10:4538 , piece_id=ZYVABAM7WRV7HCVPEAU7WS5SZERX6TDWCT5JHFBVJ2G4LURFRQUQ, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335155042 (519h25m36.206238087s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335159811 (519h25m36.206242846s) [noise=true , address=145.79.227.34:4919 , piece_id=RRCCH5ZP4VM2W6L2HZBIHD4CADDLWNPJNOGCKJSSFCVHH6ZCZMSA, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335283357 (519h25m36.206366392s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122jQwynUC2VXPahPrWNSKvLvRpHwYYtW2uQ6Fa8saDLJBPxsB5): protocol: expected piece hash; context canceled; EOF Dec 27 06:54:36.335292734 (519h25m36.206375769s) [noise=true , address=45.91.116.4:4945 , piece_id=KVN2RUBEYUPRX7B3WO5LODOV5WMHVF4SVU2T7BFWUYT7AC7T2F3A, seg_pos={0 0}, upload=455715]: Long tail cancellation due to optimal threshold Dec 27 06:54:36.335319606 (519h25m36.206402651s) [seg_pos={0 0}, upload=455715]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 06:54:36.335392224 (519h25m36.206475259s) [upload=455715]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:54:36.335397063 (519h25m36.206480098s) [upload=455715]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:54:36.626460205 (519h25m36.49754324s) [upload=455715]: Done uploading segments... Dec 27 06:54:36.626482146 (519h25m36.497565191s) [upload=455715]: Upload finished. err: <nil> Duration: 50.892056295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...................X.....X.........X.....X.................X] duration: 5.001635405s trace: https://jaeger.dev.storj.io/trace/22a3e97af1855ad4 FAIL (2025-12-27 07:00:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001663877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...........XX.XXXXX......X.XX..XXX...X.XX........X.X.XX..X..] duration: 5.935029854s trace: https://jaeger.dev.storj.io/trace/7e7d6a774ae6269e FAIL (2025-12-27 06:59:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: 5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q) context deadline exceeded ======LOGS====== 8.265525271s) [upload=455759]: Starting upload Dec 27 06:57:58.394529532 (519h28m58.265612587s) [upload=455759]: Done starting upload Dec 27 06:57:58.394635985 (519h28m58.26571905s) [upload=455759]: Uploading segments... Dec 27 06:57:58.394880952 (519h28m58.265963987s) [upload=455759]: Got next segment. Inline: false Dec 27 06:57:58.394945263 (519h28m58.266028308s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 06:57:58.394952137 (519h28m58.266035192s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 06:57:58.639815164 (519h28m58.5108982s) [seg_pos={0 0}, upload=455759]: Begin upload segment... Dec 27 06:57:58.639867253 (519h28m58.510950298s) [seg_pos={0 0}, upload=455759]: Done begin upload segment. Dec 27 06:57:58.639893263 (519h28m58.510976298s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.639963336 (519h28m58.511046381s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.640729585 (519h28m58.51181262s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641104179 (519h28m58.512187214s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641480856 (519h28m58.512563891s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.641980057 (519h28m58.513063092s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642061112 (519h28m58.513144147s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642478186 (519h28m58.513561231s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642545403 (519h28m58.513628438s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642863810 (519h28m58.513946855s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.642994188 (519h28m58.514077233s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643145426 (519h28m58.514228461s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.643540418 (519h28m58.514623473s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644160799 (519h28m58.515243835s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.644560991 (519h28m58.515644026s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:57:58.646177860 (519h28m58.517260905s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Uploading piece... Dec 27 06:58:00.354941599 (519h29m0.226024634s) [upload=455759]: Next returned nil segment Dec 27 06:58:00.354954123 (519h29m0.226037168s) [upload=455759]: Waiting for error group managing segments... Dec 27 06:58:00.638651580 (519h29m0.509734615s) [noise=true , address=145.79.227.51:4409 , piece_id=SC5DBGNUPQ6SQOXHP4A4QYFSSHRYDGHLK3HRQ5LUOTRRJ5ALPMWA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.644240012 (519h29m0.515323057s) [noise=true , address=145.79.227.15:4874 , piece_id=J2E3RWIPPXBU4BAGE7PDECTEXMI4EUMXG6VCMCPXE4MTW76M7SOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.649912613 (519h29m0.520995658s) [noise=true , address=145.79.227.46:4603 , piece_id=4MKG6PPICWEMFFQTV3J4ZNUWK3VRQDAGBTNESTG2JLUNXGARVIQA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.650586967 (519h29m0.521670012s) [noise=true , address=145.79.227.38:4977 , piece_id=WNRWPFOC5AXUP3CWEMH6GVENWP2KN6G63TIDRTES3BJ2DFLDBFSA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.651635293 (519h29m0.522718328s) [noise=true , address=145.79.227.37:4565 , piece_id=R4YGRRCCMRYDV7TCD4JE7KDMAAPKZHO2YQYCNZ54S3OKKVTITBOQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.654430717 (519h29m0.525513762s) [noise=true , address=145.79.227.35:4482 , piece_id=COILB7ZUAOFR4HZFPBYBDYKIQBERBR27NWH3JRNQYORC6WUW2WTA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.665700136 (519h29m0.536783171s) [noise=true , address=145.79.227.2:4113 , piece_id=BY6XGKMA2M3CFL42IW6MMBEJICQAC3QFLCPIUCTLW5YDBTUKU47Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.668063316 (519h29m0.539146361s) [noise=true , address=145.79.227.50:4375 , piece_id=6STW2ONHPNTZPA23ZOXG4PCI5XQUBN7V3UKU7UTTRNTBXWKDA6CA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.682023879 (519h29m0.553106914s) [noise=true , address=145.79.227.47:4265 , piece_id=YD5TIWHSNVSZLH2LL7QVZL7RXJNE3R4IO5RVQG7BXQG5AAJPPRBA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.834079984 (519h29m0.705163029s) [noise=true , address=77.90.31.19:4844 , piece_id=5KDVQ35R3LIT5RNCSBQU6KFCAPZGWWGCRP6EO575CYKVORP7RXAA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:00.927458836 (519h29m0.798541881s) [noise=true , address=45.91.116.9:4261 , piece_id=67KG5ANFLXTEVU45OMZYUA6NLKNKMBH6SLNYFPQI27VHLXGDYL6Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.197970073 (519h29m1.069053108s) [noise=true , address=45.91.116.10:4081 , piece_id=BRXMTFKCIBTFDZBQV6XKSJE6D3VM6WWQUCFHNUFVZTI6J2YIW6HQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572665028 (519h29m1.443748074s) [noise=true , address=45.91.116.14:4192 , piece_id=OTAQS73TQWD5IRZ6HEJIZ5B2YLA5AKB7JXBWGXQXF2N5XNNSMX2Q, seg_pos={0 0}, upload=455759]: Done uploading piece. err: <nil> Dec 27 06:58:01.572687861 (519h29m1.443770906s) [seg_pos={0 0}, upload=455759]: Segment reached optimal threshold of 13 pieces. Dec 27 06:58:01.572929111 (519h29m1.444012166s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 06:58:01.572938027 (519h29m1.444021073s) [noise=true , address=77.90.31.11:4555 , piece_id=XUSKZNZA73UPHJZGCYBE2USD5MOK35DNZMGOEHTL335652N2ZXSQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573021747 (519h29m1.444104793s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52416->45.91.116.12:4225: use of closed network connection Dec 27 06:58:01.573032648 (519h29m1.444115683s) [noise=true , address=45.91.116.12:4225 , piece_id=4J65ZAZPEEWW6A7FO3BCAPFJNA3YHLQZYBY466SXCBZSVGKNU7FA, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573267155 (519h29m1.4443502s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:56680->77.90.31.16:4736: use of closed network connection; context canceled Dec 27 06:58:01.573277174 (519h29m1.444360229s) [noise=true , address=77.90.31.16:4736 , piece_id=SFV4HSPBASY2ZXPFENMBZSCE47HYXYOXKXVZIYDHO3Y4XCZ6BLGQ, seg_pos={0 0}, upload=455759]: Long tail cancellation due to optimal threshold Dec 27 06:58:01.573313503 (519h29m1.444396548s) [seg_pos={0 0}, upload=455759]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 06:58:01.573408814 (519h29m1.444491859s) [upload=455759]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 06:58:01.573415266 (519h29m1.444498312s) [upload=455759]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 06:58:01.866443211 (519h29m1.737526266s) [upload=455759]: Done uploading segments... Dec 27 06:58:01.866472807 (519h29m1.737555852s) [upload=455759]: Upload finished. err: <nil> Duration: 5.935065572s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)