Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X...X..] duration: 8.7783901s trace: https://jaeger.dev.storj.io/trace/46dfd129ac474057 FAIL (2025-12-29 10:05:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:04:11.220305037 (570h35m11.091388072s) [upload=500242]: Starting upload Dec 29 10:04:11.220390911 (570h35m11.091473956s) [upload=500242]: Done starting upload Dec 29 10:04:11.220568078 (570h35m11.091651123s) [upload=500242]: Uploading segments... Dec 29 10:04:11.220724916 (570h35m11.091807961s) [upload=500242]: Got next segment. Inline: false Dec 29 10:04:11.220761436 (570h35m11.091844481s) [upload=500242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:04:11.220782486 (570h35m11.091865521s) [upload=500242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:04:11.474399335 (570h35m11.34548237s) [seg_pos={0 0}, upload=500242]: Begin upload segment... Dec 29 10:04:11.474453758 (570h35m11.345536793s) [seg_pos={0 0}, upload=500242]: Done begin upload segment. Dec 29 10:04:11.474489656 (570h35m11.345572691s) [noise=true , address=45.91.116.7:4009 , piece_id=BLYGZIXWHFSIOAMFN5NBM2PED44Y6YQQ42B47MUXFJ7D2JYB4K3A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.474605838 (570h35m11.345688883s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475217983 (570h35m11.346301018s) [noise=true , address=145.79.227.45:4229 , piece_id=B6P3WFHCQEUKIFHMHUH7UFLJS3A2LDTVEOZDIN637VAV3AH4XRXQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475646459 (570h35m11.346729494s) [noise=true , address=145.79.227.3:4151 , piece_id=RLSLU5O2CVCCVCVMY35F7FNTGZTQYBL2KHOMINX6L3Q3OA6FW5SA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475924428 (570h35m11.347007473s) [noise=true , address=145.79.227.11:4731 , piece_id=SZZRJLRE7LJDS5EGZ2KKUOBXFD5QAYJV6X6K2PMEWWR6D6OGSJ6Q, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476003289 (570h35m11.347086324s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476262662 (570h35m11.347345697s) [noise=true , address=145.79.227.16:4909 , piece_id=PAGUY35LP6B5MTJ5BMK2MZWUNXTINV66PFGXPB3AXPO6Y3IETK2Q, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476532797 (570h35m11.347615832s) [noise=true , address=45.91.116.14:4190 , piece_id=EAT3ZPGXMUOJEQVMBGRLJDIJSRXRYREPC4BIZJ5HHJIGGGJNDK5A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476786770 (570h35m11.347869815s) [noise=true , address=77.90.31.19:4844 , piece_id=SIBVBCISVESPAP3PMRJIF7C4F33OQW3WHRSOAEWKR2TCG7K2EM7A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476912069 (570h35m11.347995104s) [noise=true , address=45.91.116.9:4263 , piece_id=D24IJ7DRFAJJRLOM5M6LQ5NIP65RRCEFL5KHCDPXBE2UKWO734BQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477297333 (570h35m11.348380368s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477335866 (570h35m11.348418901s) [noise=true , address=145.79.227.38:4979 , piece_id=IXVV62TKE36RKRUQI2HCOAZ3THRZWJW4CCSEUN2PKWVQF2723OJQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477951648 (570h35m11.349034693s) [noise=true , address=145.79.227.5:4223 , piece_id=H2WYMJDOCVH6XVJZW7QCP62ZTZMQ2FCQOEAIUA67E4TPBXS2BEQQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.478507827 (570h35m11.349590862s) [noise=true , address=145.79.227.34:4906 , piece_id=NYY6WRTE5PG5DIW3B6GS75EUQKJZF4VGUJIWACXUKE5S53PSMFYQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.479139440 (570h35m11.350222475s) [noise=true , address=145.79.227.14:4839 , piece_id=RYQEX4GABVUPMAEIIUSIKOCU6SMDL4ROIHWKREPWVIT7XLYL24CQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.479662997 (570h35m11.350746042s) [noise=true , address=145.79.227.36:4519 , piece_id=TJD5PMFSJIJFXFORMN6SUV6MHDG2QPSBHT526RAPR56COQPBBNMA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:13.172511547 (570h35m13.043594582s) [upload=500242]: Next returned nil segment Dec 29 10:04:13.172526587 (570h35m13.043609632s) [upload=500242]: Waiting for error group managing segments... Dec 29 10:04:13.459650568 (570h35m13.330733603s) [noise=true , address=145.79.227.45:4229 , piece_id=B6P3WFHCQEUKIFHMHUH7UFLJS3A2LDTVEOZDIN637VAV3AH4XRXQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.459842032 (570h35m13.330925067s) [noise=true , address=145.79.227.34:4906 , piece_id=NYY6WRTE5PG5DIW3B6GS75EUQKJZF4VGUJIWACXUKE5S53PSMFYQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.461586654 (570h35m13.332669689s) [noise=true , address=145.79.227.3:4151 , piece_id=RLSLU5O2CVCCVCVMY35F7FNTGZTQYBL2KHOMINX6L3Q3OA6FW5SA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.462893132 (570h35m13.333976177s) [noise=true , address=145.79.227.11:4731 , piece_id=SZZRJLRE7LJDS5EGZ2KKUOBXFD5QAYJV6X6K2PMEWWR6D6OGSJ6Q, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.467169365 (570h35m13.3382524s) [noise=true , address=145.79.227.14:4839 , piece_id=RYQEX4GABVUPMAEIIUSIKOCU6SMDL4ROIHWKREPWVIT7XLYL24CQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.469536613 (570h35m13.340619648s) [noise=true , address=145.79.227.38:4979 , piece_id=IXVV62TKE36RKRUQI2HCOAZ3THRZWJW4CCSEUN2PKWVQF2723OJQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.476156938 (570h35m13.347239973s) [noise=true , address=145.79.227.5:4223 , piece_id=H2WYMJDOCVH6XVJZW7QCP62ZTZMQ2FCQOEAIUA67E4TPBXS2BEQQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.498921385 (570h35m13.37000442s) [noise=true , address=145.79.227.36:4519 , piece_id=TJD5PMFSJIJFXFORMN6SUV6MHDG2QPSBHT526RAPR56COQPBBNMA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.500272068 (570h35m13.371355103s) [noise=true , address=145.79.227.16:4909 , piece_id=PAGUY35LP6B5MTJ5BMK2MZWUNXTINV66PFGXPB3AXPO6Y3IETK2Q, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.699641881 (570h35m13.570724916s) [noise=true , address=45.91.116.14:4190 , piece_id=EAT3ZPGXMUOJEQVMBGRLJDIJSRXRYREPC4BIZJ5HHJIGGGJNDK5A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.764848574 (570h35m13.63593162s) [noise=true , address=77.90.31.19:4844 , piece_id=SIBVBCISVESPAP3PMRJIF7C4F33OQW3WHRSOAEWKR2TCG7K2EM7A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.768545545 (570h35m13.63962859s) [noise=true , address=45.91.116.7:4009 , piece_id=BLYGZIXWHFSIOAMFN5NBM2PED44Y6YQQ42B47MUXFJ7D2JYB4K3A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.879041758 (570h35m13.750124793s) [noise=true , address=45.91.116.9:4263 , piece_id=D24IJ7DRFAJJRLOM5M6LQ5NIP65RRCEFL5KHCDPXBE2UKWO734BQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.879063289 (570h35m13.750146334s) [seg_pos={0 0}, upload=500242]: Segment reached optimal threshold of 13 pieces. Dec 29 10:04:13.879278519 (570h35m13.750361564s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879289830 (570h35m13.750372875s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879323244 (570h35m13.750406279s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879330637 (570h35m13.750413682s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879548121 (570h35m13.750631156s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879564032 (570h35m13.750647077s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879584071 (570h35m13.750667106s) [seg_pos={0 0}, upload=500242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:04:13.879657110 (570h35m13.750740155s) [upload=500242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:04:13.879663011 (570h35m13.750746046s) [upload=500242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:04:14.178820665 (570h35m14.0499038s) [upload=500242]: Done uploading segments... Dec 29 10:04:14.178844069 (570h35m14.049927124s) [upload=500242]: Upload finished. err: <nil> Duration: 8.778410951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.....................X..................X.] duration: 32.411826081s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:05:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 32.411840427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X...X.] duration: 29.915796263s trace: https://jaeger.dev.storj.io/trace/46dfd129ac474057 FAIL (2025-12-29 10:05:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:04:11.220305037 (570h35m11.091388072s) [upload=500242]: Starting upload Dec 29 10:04:11.220390911 (570h35m11.091473956s) [upload=500242]: Done starting upload Dec 29 10:04:11.220568078 (570h35m11.091651123s) [upload=500242]: Uploading segments... Dec 29 10:04:11.220724916 (570h35m11.091807961s) [upload=500242]: Got next segment. Inline: false Dec 29 10:04:11.220761436 (570h35m11.091844481s) [upload=500242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:04:11.220782486 (570h35m11.091865521s) [upload=500242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:04:11.474399335 (570h35m11.34548237s) [seg_pos={0 0}, upload=500242]: Begin upload segment... Dec 29 10:04:11.474453758 (570h35m11.345536793s) [seg_pos={0 0}, upload=500242]: Done begin upload segment. Dec 29 10:04:11.474489656 (570h35m11.345572691s) [noise=true , address=45.91.116.7:4009 , piece_id=BLYGZIXWHFSIOAMFN5NBM2PED44Y6YQQ42B47MUXFJ7D2JYB4K3A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.474605838 (570h35m11.345688883s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475217983 (570h35m11.346301018s) [noise=true , address=145.79.227.45:4229 , piece_id=B6P3WFHCQEUKIFHMHUH7UFLJS3A2LDTVEOZDIN637VAV3AH4XRXQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475646459 (570h35m11.346729494s) [noise=true , address=145.79.227.3:4151 , piece_id=RLSLU5O2CVCCVCVMY35F7FNTGZTQYBL2KHOMINX6L3Q3OA6FW5SA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475924428 (570h35m11.347007473s) [noise=true , address=145.79.227.11:4731 , piece_id=SZZRJLRE7LJDS5EGZ2KKUOBXFD5QAYJV6X6K2PMEWWR6D6OGSJ6Q, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476003289 (570h35m11.347086324s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476262662 (570h35m11.347345697s) [noise=true , address=145.79.227.16:4909 , piece_id=PAGUY35LP6B5MTJ5BMK2MZWUNXTINV66PFGXPB3AXPO6Y3IETK2Q, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476532797 (570h35m11.347615832s) [noise=true , address=45.91.116.14:4190 , piece_id=EAT3ZPGXMUOJEQVMBGRLJDIJSRXRYREPC4BIZJ5HHJIGGGJNDK5A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476786770 (570h35m11.347869815s) [noise=true , address=77.90.31.19:4844 , piece_id=SIBVBCISVESPAP3PMRJIF7C4F33OQW3WHRSOAEWKR2TCG7K2EM7A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476912069 (570h35m11.347995104s) [noise=true , address=45.91.116.9:4263 , piece_id=D24IJ7DRFAJJRLOM5M6LQ5NIP65RRCEFL5KHCDPXBE2UKWO734BQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477297333 (570h35m11.348380368s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477335866 (570h35m11.348418901s) [noise=true , address=145.79.227.38:4979 , piece_id=IXVV62TKE36RKRUQI2HCOAZ3THRZWJW4CCSEUN2PKWVQF2723OJQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477951648 (570h35m11.349034693s) [noise=true , address=145.79.227.5:4223 , piece_id=H2WYMJDOCVH6XVJZW7QCP62ZTZMQ2FCQOEAIUA67E4TPBXS2BEQQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.478507827 (570h35m11.349590862s) [noise=true , address=145.79.227.34:4906 , piece_id=NYY6WRTE5PG5DIW3B6GS75EUQKJZF4VGUJIWACXUKE5S53PSMFYQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.479139440 (570h35m11.350222475s) [noise=true , address=145.79.227.14:4839 , piece_id=RYQEX4GABVUPMAEIIUSIKOCU6SMDL4ROIHWKREPWVIT7XLYL24CQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.479662997 (570h35m11.350746042s) [noise=true , address=145.79.227.36:4519 , piece_id=TJD5PMFSJIJFXFORMN6SUV6MHDG2QPSBHT526RAPR56COQPBBNMA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:13.172511547 (570h35m13.043594582s) [upload=500242]: Next returned nil segment Dec 29 10:04:13.172526587 (570h35m13.043609632s) [upload=500242]: Waiting for error group managing segments... Dec 29 10:04:13.459650568 (570h35m13.330733603s) [noise=true , address=145.79.227.45:4229 , piece_id=B6P3WFHCQEUKIFHMHUH7UFLJS3A2LDTVEOZDIN637VAV3AH4XRXQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.459842032 (570h35m13.330925067s) [noise=true , address=145.79.227.34:4906 , piece_id=NYY6WRTE5PG5DIW3B6GS75EUQKJZF4VGUJIWACXUKE5S53PSMFYQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.461586654 (570h35m13.332669689s) [noise=true , address=145.79.227.3:4151 , piece_id=RLSLU5O2CVCCVCVMY35F7FNTGZTQYBL2KHOMINX6L3Q3OA6FW5SA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.462893132 (570h35m13.333976177s) [noise=true , address=145.79.227.11:4731 , piece_id=SZZRJLRE7LJDS5EGZ2KKUOBXFD5QAYJV6X6K2PMEWWR6D6OGSJ6Q, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.467169365 (570h35m13.3382524s) [noise=true , address=145.79.227.14:4839 , piece_id=RYQEX4GABVUPMAEIIUSIKOCU6SMDL4ROIHWKREPWVIT7XLYL24CQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.469536613 (570h35m13.340619648s) [noise=true , address=145.79.227.38:4979 , piece_id=IXVV62TKE36RKRUQI2HCOAZ3THRZWJW4CCSEUN2PKWVQF2723OJQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.476156938 (570h35m13.347239973s) [noise=true , address=145.79.227.5:4223 , piece_id=H2WYMJDOCVH6XVJZW7QCP62ZTZMQ2FCQOEAIUA67E4TPBXS2BEQQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.498921385 (570h35m13.37000442s) [noise=true , address=145.79.227.36:4519 , piece_id=TJD5PMFSJIJFXFORMN6SUV6MHDG2QPSBHT526RAPR56COQPBBNMA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.500272068 (570h35m13.371355103s) [noise=true , address=145.79.227.16:4909 , piece_id=PAGUY35LP6B5MTJ5BMK2MZWUNXTINV66PFGXPB3AXPO6Y3IETK2Q, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.699641881 (570h35m13.570724916s) [noise=true , address=45.91.116.14:4190 , piece_id=EAT3ZPGXMUOJEQVMBGRLJDIJSRXRYREPC4BIZJ5HHJIGGGJNDK5A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.764848574 (570h35m13.63593162s) [noise=true , address=77.90.31.19:4844 , piece_id=SIBVBCISVESPAP3PMRJIF7C4F33OQW3WHRSOAEWKR2TCG7K2EM7A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.768545545 (570h35m13.63962859s) [noise=true , address=45.91.116.7:4009 , piece_id=BLYGZIXWHFSIOAMFN5NBM2PED44Y6YQQ42B47MUXFJ7D2JYB4K3A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.879041758 (570h35m13.750124793s) [noise=true , address=45.91.116.9:4263 , piece_id=D24IJ7DRFAJJRLOM5M6LQ5NIP65RRCEFL5KHCDPXBE2UKWO734BQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.879063289 (570h35m13.750146334s) [seg_pos={0 0}, upload=500242]: Segment reached optimal threshold of 13 pieces. Dec 29 10:04:13.879278519 (570h35m13.750361564s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879289830 (570h35m13.750372875s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879323244 (570h35m13.750406279s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879330637 (570h35m13.750413682s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879548121 (570h35m13.750631156s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879564032 (570h35m13.750647077s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879584071 (570h35m13.750667106s) [seg_pos={0 0}, upload=500242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:04:13.879657110 (570h35m13.750740155s) [upload=500242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:04:13.879663011 (570h35m13.750746046s) [upload=500242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:04:14.178820665 (570h35m14.0499038s) [upload=500242]: Done uploading segments... Dec 29 10:04:14.178844069 (570h35m14.049927124s) [upload=500242]: Upload finished. err: <nil> Duration: 29.915826802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.....................X..................X] duration: 1m0.001885903s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:04:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 1m0.001907123s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X...X] duration: 30.002352746s trace: https://jaeger.dev.storj.io/trace/46dfd129ac474057 FAIL (2025-12-29 10:04:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:04:11.220305037 (570h35m11.091388072s) [upload=500242]: Starting upload Dec 29 10:04:11.220390911 (570h35m11.091473956s) [upload=500242]: Done starting upload Dec 29 10:04:11.220568078 (570h35m11.091651123s) [upload=500242]: Uploading segments... Dec 29 10:04:11.220724916 (570h35m11.091807961s) [upload=500242]: Got next segment. Inline: false Dec 29 10:04:11.220761436 (570h35m11.091844481s) [upload=500242]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:04:11.220782486 (570h35m11.091865521s) [upload=500242]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:04:11.474399335 (570h35m11.34548237s) [seg_pos={0 0}, upload=500242]: Begin upload segment... Dec 29 10:04:11.474453758 (570h35m11.345536793s) [seg_pos={0 0}, upload=500242]: Done begin upload segment. Dec 29 10:04:11.474489656 (570h35m11.345572691s) [noise=true , address=45.91.116.7:4009 , piece_id=BLYGZIXWHFSIOAMFN5NBM2PED44Y6YQQ42B47MUXFJ7D2JYB4K3A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.474605838 (570h35m11.345688883s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475217983 (570h35m11.346301018s) [noise=true , address=145.79.227.45:4229 , piece_id=B6P3WFHCQEUKIFHMHUH7UFLJS3A2LDTVEOZDIN637VAV3AH4XRXQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475646459 (570h35m11.346729494s) [noise=true , address=145.79.227.3:4151 , piece_id=RLSLU5O2CVCCVCVMY35F7FNTGZTQYBL2KHOMINX6L3Q3OA6FW5SA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.475924428 (570h35m11.347007473s) [noise=true , address=145.79.227.11:4731 , piece_id=SZZRJLRE7LJDS5EGZ2KKUOBXFD5QAYJV6X6K2PMEWWR6D6OGSJ6Q, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476003289 (570h35m11.347086324s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476262662 (570h35m11.347345697s) [noise=true , address=145.79.227.16:4909 , piece_id=PAGUY35LP6B5MTJ5BMK2MZWUNXTINV66PFGXPB3AXPO6Y3IETK2Q, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476532797 (570h35m11.347615832s) [noise=true , address=45.91.116.14:4190 , piece_id=EAT3ZPGXMUOJEQVMBGRLJDIJSRXRYREPC4BIZJ5HHJIGGGJNDK5A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476786770 (570h35m11.347869815s) [noise=true , address=77.90.31.19:4844 , piece_id=SIBVBCISVESPAP3PMRJIF7C4F33OQW3WHRSOAEWKR2TCG7K2EM7A, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.476912069 (570h35m11.347995104s) [noise=true , address=45.91.116.9:4263 , piece_id=D24IJ7DRFAJJRLOM5M6LQ5NIP65RRCEFL5KHCDPXBE2UKWO734BQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477297333 (570h35m11.348380368s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477335866 (570h35m11.348418901s) [noise=true , address=145.79.227.38:4979 , piece_id=IXVV62TKE36RKRUQI2HCOAZ3THRZWJW4CCSEUN2PKWVQF2723OJQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.477951648 (570h35m11.349034693s) [noise=true , address=145.79.227.5:4223 , piece_id=H2WYMJDOCVH6XVJZW7QCP62ZTZMQ2FCQOEAIUA67E4TPBXS2BEQQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.478507827 (570h35m11.349590862s) [noise=true , address=145.79.227.34:4906 , piece_id=NYY6WRTE5PG5DIW3B6GS75EUQKJZF4VGUJIWACXUKE5S53PSMFYQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.479139440 (570h35m11.350222475s) [noise=true , address=145.79.227.14:4839 , piece_id=RYQEX4GABVUPMAEIIUSIKOCU6SMDL4ROIHWKREPWVIT7XLYL24CQ, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:11.479662997 (570h35m11.350746042s) [noise=true , address=145.79.227.36:4519 , piece_id=TJD5PMFSJIJFXFORMN6SUV6MHDG2QPSBHT526RAPR56COQPBBNMA, seg_pos={0 0}, upload=500242]: Uploading piece... Dec 29 10:04:13.172511547 (570h35m13.043594582s) [upload=500242]: Next returned nil segment Dec 29 10:04:13.172526587 (570h35m13.043609632s) [upload=500242]: Waiting for error group managing segments... Dec 29 10:04:13.459650568 (570h35m13.330733603s) [noise=true , address=145.79.227.45:4229 , piece_id=B6P3WFHCQEUKIFHMHUH7UFLJS3A2LDTVEOZDIN637VAV3AH4XRXQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.459842032 (570h35m13.330925067s) [noise=true , address=145.79.227.34:4906 , piece_id=NYY6WRTE5PG5DIW3B6GS75EUQKJZF4VGUJIWACXUKE5S53PSMFYQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.461586654 (570h35m13.332669689s) [noise=true , address=145.79.227.3:4151 , piece_id=RLSLU5O2CVCCVCVMY35F7FNTGZTQYBL2KHOMINX6L3Q3OA6FW5SA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.462893132 (570h35m13.333976177s) [noise=true , address=145.79.227.11:4731 , piece_id=SZZRJLRE7LJDS5EGZ2KKUOBXFD5QAYJV6X6K2PMEWWR6D6OGSJ6Q, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.467169365 (570h35m13.3382524s) [noise=true , address=145.79.227.14:4839 , piece_id=RYQEX4GABVUPMAEIIUSIKOCU6SMDL4ROIHWKREPWVIT7XLYL24CQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.469536613 (570h35m13.340619648s) [noise=true , address=145.79.227.38:4979 , piece_id=IXVV62TKE36RKRUQI2HCOAZ3THRZWJW4CCSEUN2PKWVQF2723OJQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.476156938 (570h35m13.347239973s) [noise=true , address=145.79.227.5:4223 , piece_id=H2WYMJDOCVH6XVJZW7QCP62ZTZMQ2FCQOEAIUA67E4TPBXS2BEQQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.498921385 (570h35m13.37000442s) [noise=true , address=145.79.227.36:4519 , piece_id=TJD5PMFSJIJFXFORMN6SUV6MHDG2QPSBHT526RAPR56COQPBBNMA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.500272068 (570h35m13.371355103s) [noise=true , address=145.79.227.16:4909 , piece_id=PAGUY35LP6B5MTJ5BMK2MZWUNXTINV66PFGXPB3AXPO6Y3IETK2Q, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.699641881 (570h35m13.570724916s) [noise=true , address=45.91.116.14:4190 , piece_id=EAT3ZPGXMUOJEQVMBGRLJDIJSRXRYREPC4BIZJ5HHJIGGGJNDK5A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.764848574 (570h35m13.63593162s) [noise=true , address=77.90.31.19:4844 , piece_id=SIBVBCISVESPAP3PMRJIF7C4F33OQW3WHRSOAEWKR2TCG7K2EM7A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.768545545 (570h35m13.63962859s) [noise=true , address=45.91.116.7:4009 , piece_id=BLYGZIXWHFSIOAMFN5NBM2PED44Y6YQQ42B47MUXFJ7D2JYB4K3A, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.879041758 (570h35m13.750124793s) [noise=true , address=45.91.116.9:4263 , piece_id=D24IJ7DRFAJJRLOM5M6LQ5NIP65RRCEFL5KHCDPXBE2UKWO734BQ, seg_pos={0 0}, upload=500242]: Done uploading piece. err: <nil> Dec 29 10:04:13.879063289 (570h35m13.750146334s) [seg_pos={0 0}, upload=500242]: Segment reached optimal threshold of 13 pieces. Dec 29 10:04:13.879278519 (570h35m13.750361564s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879289830 (570h35m13.750372875s) [noise=true , address=145.79.227.35:4482 , piece_id=DSNV5PZRAMK3LCNKQ7OEVZUCB7AAM5DIQN32M4W5SI23PPRK35DA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879323244 (570h35m13.750406279s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879330637 (570h35m13.750413682s) [noise=true , address=77.90.31.11:4553 , piece_id=PPDEAFSFFYZXFH7H34OLHZZXVGFWCOXAVFO4WPYATF5N3KUGCXOA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879548121 (570h35m13.750631156s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 29 10:04:13.879564032 (570h35m13.750647077s) [noise=true , address=145.79.227.49:4337 , piece_id=A4OG4HSBKWLCDHIUQJGB337H5EXA4MATRPCS6UODDIUAXCFLPNUA, seg_pos={0 0}, upload=500242]: Long tail cancellation due to optimal threshold Dec 29 10:04:13.879584071 (570h35m13.750667106s) [seg_pos={0 0}, upload=500242]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:04:13.879657110 (570h35m13.750740155s) [upload=500242]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:04:13.879663011 (570h35m13.750746046s) [upload=500242]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:04:14.178820665 (570h35m14.0499038s) [upload=500242]: Done uploading segments... Dec 29 10:04:14.178844069 (570h35m14.049927124s) [upload=500242]: Upload finished. err: <nil> Duration: 30.002376701s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X....................................XXX..XXX......X..X.....] duration: 3.494335849s trace: https://jaeger.dev.storj.io/trace/4fbbe44be2f4f07b FAIL (2025-12-29 10:04:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494360755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X...] duration: 5.943122737s trace: https://jaeger.dev.storj.io/trace/656af853eab165bc FAIL (2025-12-29 10:03:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A) context deadline exceeded ======LOGS====== ec 29 10:01:21.947829269 (570h32m21.818912304s) [upload=500202]: Starting upload Dec 29 10:01:21.947914461 (570h32m21.818997517s) [upload=500202]: Done starting upload Dec 29 10:01:21.948053476 (570h32m21.819136541s) [upload=500202]: Uploading segments... Dec 29 10:01:21.948279175 (570h32m21.819362231s) [upload=500202]: Got next segment. Inline: false Dec 29 10:01:21.948328159 (570h32m21.819411214s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:01:21.948335333 (570h32m21.819418378s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:01:22.193982354 (570h32m22.06506539s) [seg_pos={0 0}, upload=500202]: Begin upload segment... Dec 29 10:01:22.194028353 (570h32m22.065111388s) [seg_pos={0 0}, upload=500202]: Done begin upload segment. Dec 29 10:01:22.194068098 (570h32m22.065151143s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194076043 (570h32m22.065159088s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194603517 (570h32m22.065686552s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194669964 (570h32m22.065752999s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195077270 (570h32m22.066160315s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195615835 (570h32m22.06669887s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196113382 (570h32m22.067196417s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196312942 (570h32m22.067395987s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196746789 (570h32m22.067829824s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196841509 (570h32m22.067924544s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197394923 (570h32m22.068477958s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197461800 (570h32m22.068544845s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198249220 (570h32m22.069332255s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198366342 (570h32m22.069449377s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198677455 (570h32m22.0697605s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.200779397 (570h32m22.071862432s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:23.901104373 (570h32m23.772187419s) [upload=500202]: Next returned nil segment Dec 29 10:01:23.901129732 (570h32m23.772212767s) [upload=500202]: Waiting for error group managing segments... Dec 29 10:01:24.173047117 (570h32m24.044130162s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183128734 (570h32m24.054211779s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183911896 (570h32m24.054994931s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.184408932 (570h32m24.055491967s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.190952963 (570h32m24.062035998s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.196910988 (570h32m24.067994033s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.206600719 (570h32m24.077683754s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.207138984 (570h32m24.078222029s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.436792782 (570h32m24.307875817s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.468577265 (570h32m24.3396603s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.479145298 (570h32m24.350228333s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.488071084 (570h32m24.359154119s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529372269 (570h32m24.400455304s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529395793 (570h32m24.400478838s) [seg_pos={0 0}, upload=500202]: Segment reached optimal threshold of 13 pieces. Dec 29 10:01:24.529653865 (570h32m24.40073694s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529664094 (570h32m24.400747149s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529688631 (570h32m24.400771696s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529697027 (570h32m24.400780062s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529919440 (570h32m24.401002475s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection Dec 29 10:01:24.529936243 (570h32m24.401019298s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529958665 (570h32m24.40104171s) [seg_pos={0 0}, upload=500202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:01:24.530039609 (570h32m24.401122675s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:01:24.530046633 (570h32m24.401129688s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:01:24.821304266 (570h32m24.692387301s) [upload=500202]: Done uploading segments... Dec 29 10:01:24.821326427 (570h32m24.692409462s) [upload=500202]: Upload finished. err: <nil> Duration: 5.943148776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.....................X..................] duration: 39.792680962s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 10:03:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 39.792698967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X....................................XXX..XXX......X..X....] duration: 4.173847809s trace: https://jaeger.dev.storj.io/trace/4fbbe44be2f4f07b FAIL (2025-12-29 10:03:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.173876364s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X....................................XXX..XXX......X..X...] duration: 5.000474624s trace: https://jaeger.dev.storj.io/trace/4fbbe44be2f4f07b FAIL (2025-12-29 10:03:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000499792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X..] duration: 5.915311239s trace: https://jaeger.dev.storj.io/trace/656af853eab165bc FAIL (2025-12-29 10:02:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A) context deadline exceeded ======LOGS====== ec 29 10:01:21.947829269 (570h32m21.818912304s) [upload=500202]: Starting upload Dec 29 10:01:21.947914461 (570h32m21.818997517s) [upload=500202]: Done starting upload Dec 29 10:01:21.948053476 (570h32m21.819136541s) [upload=500202]: Uploading segments... Dec 29 10:01:21.948279175 (570h32m21.819362231s) [upload=500202]: Got next segment. Inline: false Dec 29 10:01:21.948328159 (570h32m21.819411214s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:01:21.948335333 (570h32m21.819418378s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:01:22.193982354 (570h32m22.06506539s) [seg_pos={0 0}, upload=500202]: Begin upload segment... Dec 29 10:01:22.194028353 (570h32m22.065111388s) [seg_pos={0 0}, upload=500202]: Done begin upload segment. Dec 29 10:01:22.194068098 (570h32m22.065151143s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194076043 (570h32m22.065159088s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194603517 (570h32m22.065686552s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194669964 (570h32m22.065752999s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195077270 (570h32m22.066160315s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195615835 (570h32m22.06669887s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196113382 (570h32m22.067196417s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196312942 (570h32m22.067395987s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196746789 (570h32m22.067829824s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196841509 (570h32m22.067924544s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197394923 (570h32m22.068477958s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197461800 (570h32m22.068544845s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198249220 (570h32m22.069332255s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198366342 (570h32m22.069449377s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198677455 (570h32m22.0697605s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.200779397 (570h32m22.071862432s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:23.901104373 (570h32m23.772187419s) [upload=500202]: Next returned nil segment Dec 29 10:01:23.901129732 (570h32m23.772212767s) [upload=500202]: Waiting for error group managing segments... Dec 29 10:01:24.173047117 (570h32m24.044130162s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183128734 (570h32m24.054211779s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183911896 (570h32m24.054994931s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.184408932 (570h32m24.055491967s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.190952963 (570h32m24.062035998s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.196910988 (570h32m24.067994033s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.206600719 (570h32m24.077683754s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.207138984 (570h32m24.078222029s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.436792782 (570h32m24.307875817s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.468577265 (570h32m24.3396603s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.479145298 (570h32m24.350228333s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.488071084 (570h32m24.359154119s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529372269 (570h32m24.400455304s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529395793 (570h32m24.400478838s) [seg_pos={0 0}, upload=500202]: Segment reached optimal threshold of 13 pieces. Dec 29 10:01:24.529653865 (570h32m24.40073694s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529664094 (570h32m24.400747149s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529688631 (570h32m24.400771696s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529697027 (570h32m24.400780062s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529919440 (570h32m24.401002475s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection Dec 29 10:01:24.529936243 (570h32m24.401019298s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529958665 (570h32m24.40104171s) [seg_pos={0 0}, upload=500202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:01:24.530039609 (570h32m24.401122675s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:01:24.530046633 (570h32m24.401129688s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:01:24.821304266 (570h32m24.692387301s) [upload=500202]: Done uploading segments... Dec 29 10:01:24.821326427 (570h32m24.692409462s) [upload=500202]: Upload finished. err: <nil> Duration: 5.915338723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X.] duration: 5.696201611s trace: https://jaeger.dev.storj.io/trace/656af853eab165bc FAIL (2025-12-29 10:02:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A) context deadline exceeded ======LOGS====== ec 29 10:01:21.947829269 (570h32m21.818912304s) [upload=500202]: Starting upload Dec 29 10:01:21.947914461 (570h32m21.818997517s) [upload=500202]: Done starting upload Dec 29 10:01:21.948053476 (570h32m21.819136541s) [upload=500202]: Uploading segments... Dec 29 10:01:21.948279175 (570h32m21.819362231s) [upload=500202]: Got next segment. Inline: false Dec 29 10:01:21.948328159 (570h32m21.819411214s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:01:21.948335333 (570h32m21.819418378s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:01:22.193982354 (570h32m22.06506539s) [seg_pos={0 0}, upload=500202]: Begin upload segment... Dec 29 10:01:22.194028353 (570h32m22.065111388s) [seg_pos={0 0}, upload=500202]: Done begin upload segment. Dec 29 10:01:22.194068098 (570h32m22.065151143s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194076043 (570h32m22.065159088s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194603517 (570h32m22.065686552s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194669964 (570h32m22.065752999s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195077270 (570h32m22.066160315s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195615835 (570h32m22.06669887s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196113382 (570h32m22.067196417s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196312942 (570h32m22.067395987s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196746789 (570h32m22.067829824s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196841509 (570h32m22.067924544s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197394923 (570h32m22.068477958s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197461800 (570h32m22.068544845s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198249220 (570h32m22.069332255s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198366342 (570h32m22.069449377s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198677455 (570h32m22.0697605s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.200779397 (570h32m22.071862432s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:23.901104373 (570h32m23.772187419s) [upload=500202]: Next returned nil segment Dec 29 10:01:23.901129732 (570h32m23.772212767s) [upload=500202]: Waiting for error group managing segments... Dec 29 10:01:24.173047117 (570h32m24.044130162s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183128734 (570h32m24.054211779s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183911896 (570h32m24.054994931s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.184408932 (570h32m24.055491967s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.190952963 (570h32m24.062035998s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.196910988 (570h32m24.067994033s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.206600719 (570h32m24.077683754s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.207138984 (570h32m24.078222029s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.436792782 (570h32m24.307875817s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.468577265 (570h32m24.3396603s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.479145298 (570h32m24.350228333s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.488071084 (570h32m24.359154119s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529372269 (570h32m24.400455304s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529395793 (570h32m24.400478838s) [seg_pos={0 0}, upload=500202]: Segment reached optimal threshold of 13 pieces. Dec 29 10:01:24.529653865 (570h32m24.40073694s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529664094 (570h32m24.400747149s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529688631 (570h32m24.400771696s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529697027 (570h32m24.400780062s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529919440 (570h32m24.401002475s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection Dec 29 10:01:24.529936243 (570h32m24.401019298s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529958665 (570h32m24.40104171s) [seg_pos={0 0}, upload=500202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:01:24.530039609 (570h32m24.401122675s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:01:24.530046633 (570h32m24.401129688s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:01:24.821304266 (570h32m24.692387301s) [upload=500202]: Done uploading segments... Dec 29 10:01:24.821326427 (570h32m24.692409462s) [upload=500202]: Upload finished. err: <nil> Duration: 5.696231067s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X....................................XXX..XXX......X..X..] duration: 3.424884912s trace: https://jaeger.dev.storj.io/trace/4fbbe44be2f4f07b FAIL (2025-12-29 10:02:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424903347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.....................X.................] duration: 32.506613893s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 10:02:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 32.506644493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....X] duration: 30.004896344s trace: https://jaeger.dev.storj.io/trace/656af853eab165bc FAIL (2025-12-29 10:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A) context deadline exceeded ======LOGS====== ec 29 10:01:21.947829269 (570h32m21.818912304s) [upload=500202]: Starting upload Dec 29 10:01:21.947914461 (570h32m21.818997517s) [upload=500202]: Done starting upload Dec 29 10:01:21.948053476 (570h32m21.819136541s) [upload=500202]: Uploading segments... Dec 29 10:01:21.948279175 (570h32m21.819362231s) [upload=500202]: Got next segment. Inline: false Dec 29 10:01:21.948328159 (570h32m21.819411214s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:01:21.948335333 (570h32m21.819418378s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:01:22.193982354 (570h32m22.06506539s) [seg_pos={0 0}, upload=500202]: Begin upload segment... Dec 29 10:01:22.194028353 (570h32m22.065111388s) [seg_pos={0 0}, upload=500202]: Done begin upload segment. Dec 29 10:01:22.194068098 (570h32m22.065151143s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194076043 (570h32m22.065159088s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194603517 (570h32m22.065686552s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.194669964 (570h32m22.065752999s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195077270 (570h32m22.066160315s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.195615835 (570h32m22.06669887s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196113382 (570h32m22.067196417s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196312942 (570h32m22.067395987s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196746789 (570h32m22.067829824s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.196841509 (570h32m22.067924544s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197394923 (570h32m22.068477958s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.197461800 (570h32m22.068544845s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198249220 (570h32m22.069332255s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198366342 (570h32m22.069449377s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.198677455 (570h32m22.0697605s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:22.200779397 (570h32m22.071862432s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Uploading piece... Dec 29 10:01:23.901104373 (570h32m23.772187419s) [upload=500202]: Next returned nil segment Dec 29 10:01:23.901129732 (570h32m23.772212767s) [upload=500202]: Waiting for error group managing segments... Dec 29 10:01:24.173047117 (570h32m24.044130162s) [noise=true , address=145.79.227.15:4876 , piece_id=OAPHLZ67XU63B3CHBTLMXUL2E42M35UWWVXRW2TDO3LJB2KI56YQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183128734 (570h32m24.054211779s) [noise=true , address=145.79.227.19:4078 , piece_id=3CF2SNOQY4ILWVS4O74HLAK2HEBZJT43EF5ZN4XLW4QGENRGU47A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.183911896 (570h32m24.054994931s) [noise=true , address=145.79.227.37:4567 , piece_id=ACO2OTLPKH5NELSC26YT5O33MPDZTIZU5W5XPYB5ZYT2E2GT7Y7A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.184408932 (570h32m24.055491967s) [noise=true , address=145.79.227.4:4187 , piece_id=ONZ46D5EL2ZFUWYNVU5HW4GIKB7JIILLTNPZKMB5KLFCZEVTD6CA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.190952963 (570h32m24.062035998s) [noise=true , address=145.79.227.3:4149 , piece_id=CX7J7NWL3Q2S4YBGD6NCDD6F63E2BB2WMAWQGJQGE6XH2Q4HYQUQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.196910988 (570h32m24.067994033s) [noise=true , address=145.79.227.11:4731 , piece_id=DVAACVOZNKK7R6KBKCEW4UIA5JIPUM3UTXIPIEUDMB6437HKFMDQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.206600719 (570h32m24.077683754s) [noise=true , address=145.79.227.50:4374 , piece_id=AL6UWF5OT64YGUHUZCYEPT4LOY7NINXF3ZG4ZHHGRHQI7CYXKYNA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.207138984 (570h32m24.078222029s) [noise=true , address=145.79.227.39:4013 , piece_id=5HQFYZS625X3LWMCGFUPHP5X7G6AR6WV5A7VPTXNTMU5ESDTH53A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.436792782 (570h32m24.307875817s) [noise=true , address=45.91.116.14:4190 , piece_id=Q5AS3ZDYMPZCCVVTHVO3KYOCXVBOBZHEB7DKVBR6AWQQFQWVZX5A, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.468577265 (570h32m24.3396603s) [noise=true , address=77.90.31.21:4878 , piece_id=EGSM3NKUBE5SHO2HEBEG76DPK7VX6VXU2IACZ4YACLZ3NB32ISQQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.479145298 (570h32m24.350228333s) [noise=true , address=45.91.116.4:4939 , piece_id=PVFY7KCX2IUU6RW7XV5SRJQSJAE3BNY2EU5RKT2ZMXXU2NIK3TOA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.488071084 (570h32m24.359154119s) [noise=true , address=77.90.31.12:4590 , piece_id=TUJ66RIG3H4BA6CM6JVGEAO6YRXWX4PUW7SYCGQMOWU5IGMMQHOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529372269 (570h32m24.400455304s) [noise=true , address=45.91.116.13:4155 , piece_id=4GXYIB432GJ7ADZ5SAZZZ5I4U72TXBTXWOQY5NRFIJ5JVG3DHLOQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: <nil> Dec 29 10:01:24.529395793 (570h32m24.400478838s) [seg_pos={0 0}, upload=500202]: Segment reached optimal threshold of 13 pieces. Dec 29 10:01:24.529653865 (570h32m24.40073694s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529664094 (570h32m24.400747149s) [noise=true , address=45.91.116.10:4083 , piece_id=NSHV4PYDPURFFKHCPID5DBSAV7XY4CLB5VTLZW42KYBAFGP5C2IA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529688631 (570h32m24.400771696s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 10:01:24.529697027 (570h32m24.400780062s) [noise=true , address=45.91.116.7:4011 , piece_id=4FUYOFRGFJVNAIYFKLH7FLWG4N62L5TSNMXLCBRBQU4SDIBBWNIQ, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529919440 (570h32m24.401002475s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->45.91.116.8:4047: use of closed network connection Dec 29 10:01:24.529936243 (570h32m24.401019298s) [noise=true , address=45.91.116.8:4047 , piece_id=WPQE6AENUDNCG6FWK3LED3JL5TNIC5JYB52DLGFMJSOYNVDMHNAA, seg_pos={0 0}, upload=500202]: Long tail cancellation due to optimal threshold Dec 29 10:01:24.529958665 (570h32m24.40104171s) [seg_pos={0 0}, upload=500202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:01:24.530039609 (570h32m24.401122675s) [upload=500202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:01:24.530046633 (570h32m24.401129688s) [upload=500202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:01:24.821304266 (570h32m24.692387301s) [upload=500202]: Done uploading segments... Dec 29 10:01:24.821326427 (570h32m24.692409462s) [upload=500202]: Upload finished. err: <nil> Duration: 30.0049257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X....................................XXX..XXX......X..X.] duration: 3.553315524s trace: https://jaeger.dev.storj.io/trace/4fbbe44be2f4f07b FAIL (2025-12-29 10:01:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.553342465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.....................X................] duration: 15.497303486s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 10:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 15.49732113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.....................X...............] duration: 26.167887614s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 10:00:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 26.167933813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X....] duration: 6.1317836s trace: https://jaeger.dev.storj.io/trace/4da5afa30644b8ac FAIL (2025-12-29 10:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 09:57:22.436788476 (570h28m22.307871541s) [upload=500142]: Got next segment. Inline: false Dec 29 09:57:22.436832561 (570h28m22.307915616s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:57:22.436838522 (570h28m22.307921557s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:57:22.690521565 (570h28m22.56160461s) [seg_pos={0 0}, upload=500142]: Begin upload segment... Dec 29 09:57:22.690569966 (570h28m22.561653001s) [seg_pos={0 0}, upload=500142]: Done begin upload segment. Dec 29 09:57:22.690598741 (570h28m22.561681786s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.690851183 (570h28m22.561934228s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691483326 (570h28m22.562566371s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691528232 (570h28m22.562611267s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691795501 (570h28m22.562878546s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691873058 (570h28m22.562956103s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692071927 (570h28m22.563154972s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692193809 (570h28m22.563276854s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692511323 (570h28m22.563594368s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692561710 (570h28m22.563644745s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692844147 (570h28m22.563927182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692960098 (570h28m22.564043133s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693256613 (570h28m22.564339648s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693365811 (570h28m22.564448856s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693713182 (570h28m22.564796217s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693762295 (570h28m22.56484533s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:24.296884665 (570h28m24.16796771s) [upload=500142]: Next returned nil segment Dec 29 09:57:24.296910384 (570h28m24.167993429s) [upload=500142]: Waiting for error group managing segments... Dec 29 09:57:24.679992535 (570h28m24.55107557s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.680292817 (570h28m24.551375852s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682209416 (570h28m24.553292451s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682840598 (570h28m24.553923643s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.691545774 (570h28m24.562628819s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.693241483 (570h28m24.564324518s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.698633139 (570h28m24.569716184s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702074472 (570h28m24.573157517s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702592970 (570h28m24.573676005s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.937864204 (570h28m24.808947239s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.982992396 (570h28m24.854075431s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.075805147 (570h28m24.946888182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152649251 (570h28m25.023732286s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152672384 (570h28m25.023755419s) [seg_pos={0 0}, upload=500142]: Segment reached optimal threshold of 13 pieces. Dec 29 09:57:25.152892474 (570h28m25.023975509s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection Dec 29 09:57:25.152900900 (570h28m25.023983935s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153061135 (570h28m25.02414417s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 09:57:25.153067137 (570h28m25.024150172s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153086423 (570h28m25.024169458s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection Dec 29 09:57:25.153100280 (570h28m25.024183315s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153122281 (570h28m25.024205316s) [seg_pos={0 0}, upload=500142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:57:25.153214657 (570h28m25.024297692s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:57:25.153221730 (570h28m25.024304766s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:57:25.449099610 (570h28m25.320182645s) [upload=500142]: Done uploading segments... Dec 29 09:57:25.449118487 (570h28m25.320201532s) [upload=500142]: Upload finished. err: <nil> Duration: 6.131806884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X...] duration: 27.76318578s trace: https://jaeger.dev.storj.io/trace/4da5afa30644b8ac FAIL (2025-12-29 10:00:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 09:57:22.436788476 (570h28m22.307871541s) [upload=500142]: Got next segment. Inline: false Dec 29 09:57:22.436832561 (570h28m22.307915616s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:57:22.436838522 (570h28m22.307921557s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:57:22.690521565 (570h28m22.56160461s) [seg_pos={0 0}, upload=500142]: Begin upload segment... Dec 29 09:57:22.690569966 (570h28m22.561653001s) [seg_pos={0 0}, upload=500142]: Done begin upload segment. Dec 29 09:57:22.690598741 (570h28m22.561681786s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.690851183 (570h28m22.561934228s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691483326 (570h28m22.562566371s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691528232 (570h28m22.562611267s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691795501 (570h28m22.562878546s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691873058 (570h28m22.562956103s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692071927 (570h28m22.563154972s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692193809 (570h28m22.563276854s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692511323 (570h28m22.563594368s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692561710 (570h28m22.563644745s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692844147 (570h28m22.563927182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692960098 (570h28m22.564043133s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693256613 (570h28m22.564339648s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693365811 (570h28m22.564448856s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693713182 (570h28m22.564796217s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693762295 (570h28m22.56484533s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:24.296884665 (570h28m24.16796771s) [upload=500142]: Next returned nil segment Dec 29 09:57:24.296910384 (570h28m24.167993429s) [upload=500142]: Waiting for error group managing segments... Dec 29 09:57:24.679992535 (570h28m24.55107557s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.680292817 (570h28m24.551375852s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682209416 (570h28m24.553292451s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682840598 (570h28m24.553923643s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.691545774 (570h28m24.562628819s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.693241483 (570h28m24.564324518s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.698633139 (570h28m24.569716184s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702074472 (570h28m24.573157517s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702592970 (570h28m24.573676005s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.937864204 (570h28m24.808947239s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.982992396 (570h28m24.854075431s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.075805147 (570h28m24.946888182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152649251 (570h28m25.023732286s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152672384 (570h28m25.023755419s) [seg_pos={0 0}, upload=500142]: Segment reached optimal threshold of 13 pieces. Dec 29 09:57:25.152892474 (570h28m25.023975509s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection Dec 29 09:57:25.152900900 (570h28m25.023983935s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153061135 (570h28m25.02414417s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 09:57:25.153067137 (570h28m25.024150172s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153086423 (570h28m25.024169458s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection Dec 29 09:57:25.153100280 (570h28m25.024183315s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153122281 (570h28m25.024205316s) [seg_pos={0 0}, upload=500142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:57:25.153214657 (570h28m25.024297692s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:57:25.153221730 (570h28m25.024304766s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:57:25.449099610 (570h28m25.320182645s) [upload=500142]: Done uploading segments... Dec 29 09:57:25.449118487 (570h28m25.320201532s) [upload=500142]: Upload finished. err: <nil> Duration: 27.763214273s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X....................................XXX..XXX......X..X] duration: 5.001643619s trace: https://jaeger.dev.storj.io/trace/4fbbe44be2f4f07b FAIL (2025-12-29 10:00:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001670972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X..] duration: 5.666021983s trace: https://jaeger.dev.storj.io/trace/4da5afa30644b8ac FAIL (2025-12-29 09:59:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 09:57:22.436788476 (570h28m22.307871541s) [upload=500142]: Got next segment. Inline: false Dec 29 09:57:22.436832561 (570h28m22.307915616s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:57:22.436838522 (570h28m22.307921557s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:57:22.690521565 (570h28m22.56160461s) [seg_pos={0 0}, upload=500142]: Begin upload segment... Dec 29 09:57:22.690569966 (570h28m22.561653001s) [seg_pos={0 0}, upload=500142]: Done begin upload segment. Dec 29 09:57:22.690598741 (570h28m22.561681786s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.690851183 (570h28m22.561934228s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691483326 (570h28m22.562566371s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691528232 (570h28m22.562611267s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691795501 (570h28m22.562878546s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691873058 (570h28m22.562956103s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692071927 (570h28m22.563154972s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692193809 (570h28m22.563276854s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692511323 (570h28m22.563594368s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692561710 (570h28m22.563644745s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692844147 (570h28m22.563927182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692960098 (570h28m22.564043133s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693256613 (570h28m22.564339648s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693365811 (570h28m22.564448856s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693713182 (570h28m22.564796217s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693762295 (570h28m22.56484533s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:24.296884665 (570h28m24.16796771s) [upload=500142]: Next returned nil segment Dec 29 09:57:24.296910384 (570h28m24.167993429s) [upload=500142]: Waiting for error group managing segments... Dec 29 09:57:24.679992535 (570h28m24.55107557s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.680292817 (570h28m24.551375852s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682209416 (570h28m24.553292451s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682840598 (570h28m24.553923643s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.691545774 (570h28m24.562628819s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.693241483 (570h28m24.564324518s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.698633139 (570h28m24.569716184s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702074472 (570h28m24.573157517s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702592970 (570h28m24.573676005s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.937864204 (570h28m24.808947239s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.982992396 (570h28m24.854075431s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.075805147 (570h28m24.946888182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152649251 (570h28m25.023732286s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152672384 (570h28m25.023755419s) [seg_pos={0 0}, upload=500142]: Segment reached optimal threshold of 13 pieces. Dec 29 09:57:25.152892474 (570h28m25.023975509s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection Dec 29 09:57:25.152900900 (570h28m25.023983935s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153061135 (570h28m25.02414417s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 09:57:25.153067137 (570h28m25.024150172s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153086423 (570h28m25.024169458s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection Dec 29 09:57:25.153100280 (570h28m25.024183315s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153122281 (570h28m25.024205316s) [seg_pos={0 0}, upload=500142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:57:25.153214657 (570h28m25.024297692s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:57:25.153221730 (570h28m25.024304766s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:57:25.449099610 (570h28m25.320182645s) [upload=500142]: Done uploading segments... Dec 29 09:57:25.449118487 (570h28m25.320201532s) [upload=500142]: Upload finished. err: <nil> Duration: 5.666163202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.....................X..............] duration: 15.333639258s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:59:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 15.333657433s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X....................................XXX..XXX......X..] duration: 4.890363441s trace: https://jaeger.dev.storj.io/trace/25fcd32e5cc6b190 FAIL (2025-12-29 09:59:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.890387337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.....................X.............] duration: 18.675374437s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:59:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 18.67543409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X.] duration: 5.809728002s trace: https://jaeger.dev.storj.io/trace/4da5afa30644b8ac FAIL (2025-12-29 09:58:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 09:57:22.436788476 (570h28m22.307871541s) [upload=500142]: Got next segment. Inline: false Dec 29 09:57:22.436832561 (570h28m22.307915616s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:57:22.436838522 (570h28m22.307921557s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:57:22.690521565 (570h28m22.56160461s) [seg_pos={0 0}, upload=500142]: Begin upload segment... Dec 29 09:57:22.690569966 (570h28m22.561653001s) [seg_pos={0 0}, upload=500142]: Done begin upload segment. Dec 29 09:57:22.690598741 (570h28m22.561681786s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.690851183 (570h28m22.561934228s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691483326 (570h28m22.562566371s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691528232 (570h28m22.562611267s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691795501 (570h28m22.562878546s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691873058 (570h28m22.562956103s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692071927 (570h28m22.563154972s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692193809 (570h28m22.563276854s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692511323 (570h28m22.563594368s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692561710 (570h28m22.563644745s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692844147 (570h28m22.563927182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692960098 (570h28m22.564043133s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693256613 (570h28m22.564339648s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693365811 (570h28m22.564448856s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693713182 (570h28m22.564796217s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693762295 (570h28m22.56484533s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:24.296884665 (570h28m24.16796771s) [upload=500142]: Next returned nil segment Dec 29 09:57:24.296910384 (570h28m24.167993429s) [upload=500142]: Waiting for error group managing segments... Dec 29 09:57:24.679992535 (570h28m24.55107557s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.680292817 (570h28m24.551375852s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682209416 (570h28m24.553292451s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682840598 (570h28m24.553923643s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.691545774 (570h28m24.562628819s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.693241483 (570h28m24.564324518s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.698633139 (570h28m24.569716184s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702074472 (570h28m24.573157517s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702592970 (570h28m24.573676005s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.937864204 (570h28m24.808947239s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.982992396 (570h28m24.854075431s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.075805147 (570h28m24.946888182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152649251 (570h28m25.023732286s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152672384 (570h28m25.023755419s) [seg_pos={0 0}, upload=500142]: Segment reached optimal threshold of 13 pieces. Dec 29 09:57:25.152892474 (570h28m25.023975509s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection Dec 29 09:57:25.152900900 (570h28m25.023983935s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153061135 (570h28m25.02414417s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 09:57:25.153067137 (570h28m25.024150172s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153086423 (570h28m25.024169458s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection Dec 29 09:57:25.153100280 (570h28m25.024183315s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153122281 (570h28m25.024205316s) [seg_pos={0 0}, upload=500142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:57:25.153214657 (570h28m25.024297692s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:57:25.153221730 (570h28m25.024304766s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:57:25.449099610 (570h28m25.320182645s) [upload=500142]: Done uploading segments... Dec 29 09:57:25.449118487 (570h28m25.320201532s) [upload=500142]: Upload finished. err: <nil> Duration: 5.809753189s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......X....................................XXX..XXX......X.] duration: 3.972805912s trace: https://jaeger.dev.storj.io/trace/25fcd32e5cc6b190 FAIL (2025-12-29 09:58:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.972828725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.....................X............] duration: 44.887146778s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:58:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 44.887172877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.......X....................................XXX..XXX......X] duration: 5.000732384s trace: https://jaeger.dev.storj.io/trace/25fcd32e5cc6b190 FAIL (2025-12-29 09:58:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000757001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..X] duration: 30.003489531s trace: https://jaeger.dev.storj.io/trace/4da5afa30644b8ac FAIL (2025-12-29 09:57:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 09:57:22.436788476 (570h28m22.307871541s) [upload=500142]: Got next segment. Inline: false Dec 29 09:57:22.436832561 (570h28m22.307915616s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:57:22.436838522 (570h28m22.307921557s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:57:22.690521565 (570h28m22.56160461s) [seg_pos={0 0}, upload=500142]: Begin upload segment... Dec 29 09:57:22.690569966 (570h28m22.561653001s) [seg_pos={0 0}, upload=500142]: Done begin upload segment. Dec 29 09:57:22.690598741 (570h28m22.561681786s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.690851183 (570h28m22.561934228s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691483326 (570h28m22.562566371s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691528232 (570h28m22.562611267s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691795501 (570h28m22.562878546s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.691873058 (570h28m22.562956103s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692071927 (570h28m22.563154972s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692193809 (570h28m22.563276854s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692511323 (570h28m22.563594368s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692561710 (570h28m22.563644745s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692844147 (570h28m22.563927182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.692960098 (570h28m22.564043133s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693256613 (570h28m22.564339648s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693365811 (570h28m22.564448856s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693713182 (570h28m22.564796217s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:22.693762295 (570h28m22.56484533s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Uploading piece... Dec 29 09:57:24.296884665 (570h28m24.16796771s) [upload=500142]: Next returned nil segment Dec 29 09:57:24.296910384 (570h28m24.167993429s) [upload=500142]: Waiting for error group managing segments... Dec 29 09:57:24.679992535 (570h28m24.55107557s) [noise=true , address=145.79.227.23:4545 , piece_id=QTERHHTDRGEQQA4NCYFFDUUAS5TZXQHLUHKSKZXV7YPONCO525XA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.680292817 (570h28m24.551375852s) [noise=true , address=145.79.227.37:4567 , piece_id=GFFJLIE2I3O4DGY6MRYJ2PRDNHOQNBSVQ6SR2GAJAPW3ZGNG6JZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682209416 (570h28m24.553292451s) [noise=true , address=145.79.227.45:4229 , piece_id=RIIMUIHPP2CPVTQRH632M5AFTWLY4JEKODKBHJJNWZMQS75E3AZQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.682840598 (570h28m24.553923643s) [noise=true , address=145.79.227.4:4186 , piece_id=ZWYTQVT25RHTGONODHQHAFGCONGAUKD45XQCGR72Z3OKT7L3A4BQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.691545774 (570h28m24.562628819s) [noise=true , address=145.79.227.52:4445 , piece_id=RX2LOWZAWD5HS2ELKEIVLRCGWPI6OKZ6SH2JPE5ETYNONXGH2FNA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.693241483 (570h28m24.564324518s) [noise=true , address=145.79.227.24:4582 , piece_id=FEQISAEBRVTSIUBY2PL33SY5W42HXPJ4CECEBBOS2H2M7O2HGPQQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.698633139 (570h28m24.569716184s) [noise=true , address=145.79.227.41:4087 , piece_id=25FDVWTP3DBZKA3OGN56W67V5RBAQNFBXD6TYO2YUO4XP2FC4RSQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702074472 (570h28m24.573157517s) [noise=true , address=145.79.227.11:4729 , piece_id=CNS33YYYISXVXXQHYEM2ZO5JINUPGBFVGEBO4ME6ASL6NO3AEWBA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.702592970 (570h28m24.573676005s) [noise=true , address=145.79.227.13:4802 , piece_id=TTN3YVCTUIW5LOG5RYUMOCGWT42RGFQ56Z2OJMCAJRRBTGZDPHYQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.937864204 (570h28m24.808947239s) [noise=true , address=45.91.116.6:4974 , piece_id=JZZM7W6HFRCUKNDAZUEALGRDYOW3UHV2YKAGTH6W7FF42PYKXYLA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:24.982992396 (570h28m24.854075431s) [noise=true , address=77.90.31.11:4555 , piece_id=OBFHEEXR65XVOS66I7EIGO5AACDKGNXTGFDIGOYV6FBQ6JTIIBXA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.075805147 (570h28m24.946888182s) [noise=true , address=45.91.116.13:4155 , piece_id=GL23R276NSFDCLQRQ7KF6KUSIPRTAQLM2NBDBDQHS4A7SFLX4FHA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152649251 (570h28m25.023732286s) [noise=true , address=45.91.116.5:4297 , piece_id=GMDV55XR5IZ5YPPOOWFA4VC7HFQEJHBEO6ZEYCEAMK7VB4N4GKRA, seg_pos={0 0}, upload=500142]: Done uploading piece. err: <nil> Dec 29 09:57:25.152672384 (570h28m25.023755419s) [seg_pos={0 0}, upload=500142]: Segment reached optimal threshold of 13 pieces. Dec 29 09:57:25.152892474 (570h28m25.023975509s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38692->45.91.116.11:4119: use of closed network connection Dec 29 09:57:25.152900900 (570h28m25.023983935s) [noise=true , address=45.91.116.11:4119 , piece_id=MJRDPUP3FXOUKC223VEWXOC63C66IPPGL3SEIKCIGQQDCEGKC3OQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153061135 (570h28m25.02414417s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 29 09:57:25.153067137 (570h28m25.024150172s) [noise=true , address=45.91.116.10:4082 , piece_id=TPNTFNH3YVFPHSEEVELEOS664TEG7ZSKZ24B5BH4PUVZME263DVQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153086423 (570h28m25.024169458s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57488->45.91.116.9:4264: use of closed network connection Dec 29 09:57:25.153100280 (570h28m25.024183315s) [noise=true , address=45.91.116.9:4264 , piece_id=HFCZZESOGSOU6LA2XUEVCHJAPUUPVIFHMIYKOCTARP5FYDCHGEBQ, seg_pos={0 0}, upload=500142]: Long tail cancellation due to optimal threshold Dec 29 09:57:25.153122281 (570h28m25.024205316s) [seg_pos={0 0}, upload=500142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:57:25.153214657 (570h28m25.024297692s) [upload=500142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:57:25.153221730 (570h28m25.024304766s) [upload=500142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:57:25.449099610 (570h28m25.320182645s) [upload=500142]: Done uploading segments... Dec 29 09:57:25.449118487 (570h28m25.320201532s) [upload=500142]: Upload finished. err: <nil> Duration: 30.003518376s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.......X....................................XXX..XXX......] duration: 3.422122191s trace: https://jaeger.dev.storj.io/trace/742624afa397e34c FAIL (2025-12-29 09:57:24 UTC) orbiter warning: context deadline exceeded Duration: 3.422146508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.....................X...........] duration: 35.155851611s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:57:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 35.155865898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X..] duration: 6.01920913s trace: https://jaeger.dev.storj.io/trace/5c9c8707cf3f5953 FAIL (2025-12-29 09:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA) context deadline exceeded ======LOGS====== Dec 29 09:55:48.067404478 (570h26m47.938487513s) [upload=500121]: Starting upload Dec 29 09:55:48.067491553 (570h26m47.938574608s) [upload=500121]: Done starting upload Dec 29 09:55:48.067527652 (570h26m47.938610687s) [upload=500121]: Uploading segments... Dec 29 09:55:48.067736079 (570h26m47.938819114s) [upload=500121]: Got next segment. Inline: false Dec 29 09:55:48.067769202 (570h26m47.938852237s) [upload=500121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:55:48.067774121 (570h26m47.938857156s) [upload=500121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:55:48.313041902 (570h26m48.184124937s) [seg_pos={0 0}, upload=500121]: Begin upload segment... Dec 29 09:55:48.313088370 (570h26m48.184171435s) [seg_pos={0 0}, upload=500121]: Done begin upload segment. Dec 29 09:55:48.313118888 (570h26m48.184201923s) [noise=true , address=45.91.116.14:4192 , piece_id=JX5MVZOVMEQKQUFL6KLW7YPIU4FWZ6UVBH7RP3YMNVHN3TMOMIJA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313139627 (570h26m48.184222662s) [noise=true , address=45.91.116.8:4046 , piece_id=TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313620292 (570h26m48.184703328s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313769577 (570h26m48.184852612s) [noise=true , address=145.79.227.41:4087 , piece_id=QHFWEZBDIACT53C7I37R55RXI3Y2ZDGR56P7N7B5KMVD6KRKIWEQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314090899 (570h26m48.185173934s) [noise=true , address=145.79.227.6:4258 , piece_id=75ZAI4PQXC4ALQ6XR3DQAFJ3ESZIQCO6GQ6EAZNEU6TSOC7GQKGQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314460452 (570h26m48.185543487s) [noise=true , address=145.79.227.39:4013 , piece_id=TPSFBRML57UXRNPBVWSC3VCXVFSKOUZHCJRVPBZK3SZ2HHXZVMDA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314636307 (570h26m48.185719342s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314757208 (570h26m48.185840253s) [noise=true , address=45.91.116.13:4153 , piece_id=4DVDHFYX4BVL7JTFIRGAU5EFWNBFWQE6YJ3N55VN4X5LZHH2WUJA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314969662 (570h26m48.186052697s) [noise=true , address=77.90.31.19:4844 , piece_id=MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315001212 (570h26m48.186084247s) [noise=true , address=45.91.116.11:4120 , piece_id=QNPCJXQUWA72KXMVRJMLTR3EWG7KTSWTINF43ZA5SDEUZ7VVRFUA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315319638 (570h26m48.186402684s) [noise=true , address=77.90.31.11:4555 , piece_id=ARLBME6S2RPXORLKMR3G4LE3NFTEISO3Y6OWM3SBUICX5322GWKQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315499781 (570h26m48.186582816s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315688461 (570h26m48.186771506s) [noise=true , address=77.90.31.13:4626 , piece_id=KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316006316 (570h26m48.187089351s) [noise=true , address=145.79.227.38:4978 , piece_id=OZH3XZGQW52OFES45MJYVOKMNFAVULIPXGECYADHWD4HDPNOC5CQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316309383 (570h26m48.187392418s) [noise=true , address=145.79.227.11:4729 , piece_id=OVDGHBB33RRLVZ3ONAJP5GICGP6PB3FVWX7JUJQFQJN5OJ4TMGEA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316665091 (570h26m48.187748136s) [noise=true , address=77.90.31.10:4517 , piece_id=DIAMXFJZYC3BJIRBLAC4EAJ5WXCIBDG557PDLPK5DQGHMKCMFF7A, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:50.005239292 (570h26m49.876322337s) [upload=500121]: Next returned nil segment Dec 29 09:55:50.005291762 (570h26m49.876374797s) [upload=500121]: Waiting for error group managing segments... Dec 29 09:55:50.298555816 (570h26m50.169638861s) [noise=true , address=145.79.227.39:4013 , piece_id=TPSFBRML57UXRNPBVWSC3VCXVFSKOUZHCJRVPBZK3SZ2HHXZVMDA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.306649337 (570h26m50.177732372s) [noise=true , address=145.79.227.41:4087 , piece_id=QHFWEZBDIACT53C7I37R55RXI3Y2ZDGR56P7N7B5KMVD6KRKIWEQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.318850028 (570h26m50.189933073s) [noise=true , address=145.79.227.38:4978 , piece_id=OZH3XZGQW52OFES45MJYVOKMNFAVULIPXGECYADHWD4HDPNOC5CQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.322765935 (570h26m50.19384898s) [noise=true , address=145.79.227.11:4729 , piece_id=OVDGHBB33RRLVZ3ONAJP5GICGP6PB3FVWX7JUJQFQJN5OJ4TMGEA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.324301569 (570h26m50.195384604s) [noise=true , address=145.79.227.6:4258 , piece_id=75ZAI4PQXC4ALQ6XR3DQAFJ3ESZIQCO6GQ6EAZNEU6TSOC7GQKGQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.496708931 (570h26m50.367791966s) [noise=true , address=77.90.31.10:4517 , piece_id=DIAMXFJZYC3BJIRBLAC4EAJ5WXCIBDG557PDLPK5DQGHMKCMFF7A, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.497362305 (570h26m50.36844534s) [noise=true , address=77.90.31.11:4555 , piece_id=ARLBME6S2RPXORLKMR3G4LE3NFTEISO3Y6OWM3SBUICX5322GWKQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.583064827 (570h26m50.454147872s) [noise=true , address=77.90.31.19:4844 , piece_id=MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.626062862 (570h26m50.497145897s) [noise=true , address=77.90.31.13:4626 , piece_id=KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.632464392 (570h26m50.503547437s) [noise=true , address=45.91.116.13:4153 , piece_id=4DVDHFYX4BVL7JTFIRGAU5EFWNBFWQE6YJ3N55VN4X5LZHH2WUJA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.703846945 (570h26m50.57492999s) [noise=true , address=45.91.116.14:4192 , piece_id=JX5MVZOVMEQKQUFL6KLW7YPIU4FWZ6UVBH7RP3YMNVHN3TMOMIJA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.734108439 (570h26m50.605191484s) [noise=true , address=45.91.116.11:4120 , piece_id=QNPCJXQUWA72KXMVRJMLTR3EWG7KTSWTINF43ZA5SDEUZ7VVRFUA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:51.427016850 (570h26m51.298099885s) [noise=true , address=45.91.116.8:4046 , piece_id=TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:51.427039252 (570h26m51.298122287s) [seg_pos={0 0}, upload=500121]: Segment reached optimal threshold of 13 pieces. Dec 29 09:55:51.427267056 (570h26m51.298350122s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.427278969 (570h26m51.298362014s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.428909343 (570h26m51.299992389s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.428920104 (570h26m51.300003149s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.429082433 (570h26m51.300165478s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.429090428 (570h26m51.300173473s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.429113903 (570h26m51.300196968s) [seg_pos={0 0}, upload=500121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:55:51.429194486 (570h26m51.300277531s) [upload=500121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:55:51.429201420 (570h26m51.300284455s) [upload=500121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:55:51.717085218 (570h26m51.588168253s) [upload=500121]: Done uploading segments... Dec 29 09:55:51.717101930 (570h26m51.588184975s) [upload=500121]: Upload finished. err: <nil> Duration: 6.019237663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X.] duration: 6.020762567s trace: https://jaeger.dev.storj.io/trace/5c9c8707cf3f5953 FAIL (2025-12-29 09:56:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA) context deadline exceeded ======LOGS====== Dec 29 09:55:48.067404478 (570h26m47.938487513s) [upload=500121]: Starting upload Dec 29 09:55:48.067491553 (570h26m47.938574608s) [upload=500121]: Done starting upload Dec 29 09:55:48.067527652 (570h26m47.938610687s) [upload=500121]: Uploading segments... Dec 29 09:55:48.067736079 (570h26m47.938819114s) [upload=500121]: Got next segment. Inline: false Dec 29 09:55:48.067769202 (570h26m47.938852237s) [upload=500121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:55:48.067774121 (570h26m47.938857156s) [upload=500121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:55:48.313041902 (570h26m48.184124937s) [seg_pos={0 0}, upload=500121]: Begin upload segment... Dec 29 09:55:48.313088370 (570h26m48.184171435s) [seg_pos={0 0}, upload=500121]: Done begin upload segment. Dec 29 09:55:48.313118888 (570h26m48.184201923s) [noise=true , address=45.91.116.14:4192 , piece_id=JX5MVZOVMEQKQUFL6KLW7YPIU4FWZ6UVBH7RP3YMNVHN3TMOMIJA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313139627 (570h26m48.184222662s) [noise=true , address=45.91.116.8:4046 , piece_id=TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313620292 (570h26m48.184703328s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313769577 (570h26m48.184852612s) [noise=true , address=145.79.227.41:4087 , piece_id=QHFWEZBDIACT53C7I37R55RXI3Y2ZDGR56P7N7B5KMVD6KRKIWEQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314090899 (570h26m48.185173934s) [noise=true , address=145.79.227.6:4258 , piece_id=75ZAI4PQXC4ALQ6XR3DQAFJ3ESZIQCO6GQ6EAZNEU6TSOC7GQKGQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314460452 (570h26m48.185543487s) [noise=true , address=145.79.227.39:4013 , piece_id=TPSFBRML57UXRNPBVWSC3VCXVFSKOUZHCJRVPBZK3SZ2HHXZVMDA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314636307 (570h26m48.185719342s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314757208 (570h26m48.185840253s) [noise=true , address=45.91.116.13:4153 , piece_id=4DVDHFYX4BVL7JTFIRGAU5EFWNBFWQE6YJ3N55VN4X5LZHH2WUJA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314969662 (570h26m48.186052697s) [noise=true , address=77.90.31.19:4844 , piece_id=MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315001212 (570h26m48.186084247s) [noise=true , address=45.91.116.11:4120 , piece_id=QNPCJXQUWA72KXMVRJMLTR3EWG7KTSWTINF43ZA5SDEUZ7VVRFUA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315319638 (570h26m48.186402684s) [noise=true , address=77.90.31.11:4555 , piece_id=ARLBME6S2RPXORLKMR3G4LE3NFTEISO3Y6OWM3SBUICX5322GWKQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315499781 (570h26m48.186582816s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315688461 (570h26m48.186771506s) [noise=true , address=77.90.31.13:4626 , piece_id=KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316006316 (570h26m48.187089351s) [noise=true , address=145.79.227.38:4978 , piece_id=OZH3XZGQW52OFES45MJYVOKMNFAVULIPXGECYADHWD4HDPNOC5CQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316309383 (570h26m48.187392418s) [noise=true , address=145.79.227.11:4729 , piece_id=OVDGHBB33RRLVZ3ONAJP5GICGP6PB3FVWX7JUJQFQJN5OJ4TMGEA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316665091 (570h26m48.187748136s) [noise=true , address=77.90.31.10:4517 , piece_id=DIAMXFJZYC3BJIRBLAC4EAJ5WXCIBDG557PDLPK5DQGHMKCMFF7A, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:50.005239292 (570h26m49.876322337s) [upload=500121]: Next returned nil segment Dec 29 09:55:50.005291762 (570h26m49.876374797s) [upload=500121]: Waiting for error group managing segments... Dec 29 09:55:50.298555816 (570h26m50.169638861s) [noise=true , address=145.79.227.39:4013 , piece_id=TPSFBRML57UXRNPBVWSC3VCXVFSKOUZHCJRVPBZK3SZ2HHXZVMDA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.306649337 (570h26m50.177732372s) [noise=true , address=145.79.227.41:4087 , piece_id=QHFWEZBDIACT53C7I37R55RXI3Y2ZDGR56P7N7B5KMVD6KRKIWEQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.318850028 (570h26m50.189933073s) [noise=true , address=145.79.227.38:4978 , piece_id=OZH3XZGQW52OFES45MJYVOKMNFAVULIPXGECYADHWD4HDPNOC5CQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.322765935 (570h26m50.19384898s) [noise=true , address=145.79.227.11:4729 , piece_id=OVDGHBB33RRLVZ3ONAJP5GICGP6PB3FVWX7JUJQFQJN5OJ4TMGEA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.324301569 (570h26m50.195384604s) [noise=true , address=145.79.227.6:4258 , piece_id=75ZAI4PQXC4ALQ6XR3DQAFJ3ESZIQCO6GQ6EAZNEU6TSOC7GQKGQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.496708931 (570h26m50.367791966s) [noise=true , address=77.90.31.10:4517 , piece_id=DIAMXFJZYC3BJIRBLAC4EAJ5WXCIBDG557PDLPK5DQGHMKCMFF7A, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.497362305 (570h26m50.36844534s) [noise=true , address=77.90.31.11:4555 , piece_id=ARLBME6S2RPXORLKMR3G4LE3NFTEISO3Y6OWM3SBUICX5322GWKQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.583064827 (570h26m50.454147872s) [noise=true , address=77.90.31.19:4844 , piece_id=MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.626062862 (570h26m50.497145897s) [noise=true , address=77.90.31.13:4626 , piece_id=KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.632464392 (570h26m50.503547437s) [noise=true , address=45.91.116.13:4153 , piece_id=4DVDHFYX4BVL7JTFIRGAU5EFWNBFWQE6YJ3N55VN4X5LZHH2WUJA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.703846945 (570h26m50.57492999s) [noise=true , address=45.91.116.14:4192 , piece_id=JX5MVZOVMEQKQUFL6KLW7YPIU4FWZ6UVBH7RP3YMNVHN3TMOMIJA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.734108439 (570h26m50.605191484s) [noise=true , address=45.91.116.11:4120 , piece_id=QNPCJXQUWA72KXMVRJMLTR3EWG7KTSWTINF43ZA5SDEUZ7VVRFUA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:51.427016850 (570h26m51.298099885s) [noise=true , address=45.91.116.8:4046 , piece_id=TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:51.427039252 (570h26m51.298122287s) [seg_pos={0 0}, upload=500121]: Segment reached optimal threshold of 13 pieces. Dec 29 09:55:51.427267056 (570h26m51.298350122s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.427278969 (570h26m51.298362014s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.428909343 (570h26m51.299992389s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.428920104 (570h26m51.300003149s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.429082433 (570h26m51.300165478s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.429090428 (570h26m51.300173473s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.429113903 (570h26m51.300196968s) [seg_pos={0 0}, upload=500121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:55:51.429194486 (570h26m51.300277531s) [upload=500121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:55:51.429201420 (570h26m51.300284455s) [upload=500121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:55:51.717085218 (570h26m51.588168253s) [upload=500121]: Done uploading segments... Dec 29 09:55:51.717101930 (570h26m51.588184975s) [upload=500121]: Upload finished. err: <nil> Duration: 6.020781772s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.....................X..........] duration: 53.089916281s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 53.089980602s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X.......X....................................XXX..XXX.....] duration: 3.445078626s trace: https://jaeger.dev.storj.io/trace/742624afa397e34c FAIL (2025-12-29 09:56:23 UTC) orbiter warning: context deadline exceeded Duration: 3.445192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...X] duration: 30.003650269s trace: https://jaeger.dev.storj.io/trace/5c9c8707cf3f5953 FAIL (2025-12-29 09:56:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA) context deadline exceeded ======LOGS====== Dec 29 09:55:48.067404478 (570h26m47.938487513s) [upload=500121]: Starting upload Dec 29 09:55:48.067491553 (570h26m47.938574608s) [upload=500121]: Done starting upload Dec 29 09:55:48.067527652 (570h26m47.938610687s) [upload=500121]: Uploading segments... Dec 29 09:55:48.067736079 (570h26m47.938819114s) [upload=500121]: Got next segment. Inline: false Dec 29 09:55:48.067769202 (570h26m47.938852237s) [upload=500121]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:55:48.067774121 (570h26m47.938857156s) [upload=500121]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:55:48.313041902 (570h26m48.184124937s) [seg_pos={0 0}, upload=500121]: Begin upload segment... Dec 29 09:55:48.313088370 (570h26m48.184171435s) [seg_pos={0 0}, upload=500121]: Done begin upload segment. Dec 29 09:55:48.313118888 (570h26m48.184201923s) [noise=true , address=45.91.116.14:4192 , piece_id=JX5MVZOVMEQKQUFL6KLW7YPIU4FWZ6UVBH7RP3YMNVHN3TMOMIJA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313139627 (570h26m48.184222662s) [noise=true , address=45.91.116.8:4046 , piece_id=TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313620292 (570h26m48.184703328s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.313769577 (570h26m48.184852612s) [noise=true , address=145.79.227.41:4087 , piece_id=QHFWEZBDIACT53C7I37R55RXI3Y2ZDGR56P7N7B5KMVD6KRKIWEQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314090899 (570h26m48.185173934s) [noise=true , address=145.79.227.6:4258 , piece_id=75ZAI4PQXC4ALQ6XR3DQAFJ3ESZIQCO6GQ6EAZNEU6TSOC7GQKGQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314460452 (570h26m48.185543487s) [noise=true , address=145.79.227.39:4013 , piece_id=TPSFBRML57UXRNPBVWSC3VCXVFSKOUZHCJRVPBZK3SZ2HHXZVMDA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314636307 (570h26m48.185719342s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314757208 (570h26m48.185840253s) [noise=true , address=45.91.116.13:4153 , piece_id=4DVDHFYX4BVL7JTFIRGAU5EFWNBFWQE6YJ3N55VN4X5LZHH2WUJA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.314969662 (570h26m48.186052697s) [noise=true , address=77.90.31.19:4844 , piece_id=MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315001212 (570h26m48.186084247s) [noise=true , address=45.91.116.11:4120 , piece_id=QNPCJXQUWA72KXMVRJMLTR3EWG7KTSWTINF43ZA5SDEUZ7VVRFUA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315319638 (570h26m48.186402684s) [noise=true , address=77.90.31.11:4555 , piece_id=ARLBME6S2RPXORLKMR3G4LE3NFTEISO3Y6OWM3SBUICX5322GWKQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315499781 (570h26m48.186582816s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.315688461 (570h26m48.186771506s) [noise=true , address=77.90.31.13:4626 , piece_id=KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316006316 (570h26m48.187089351s) [noise=true , address=145.79.227.38:4978 , piece_id=OZH3XZGQW52OFES45MJYVOKMNFAVULIPXGECYADHWD4HDPNOC5CQ, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316309383 (570h26m48.187392418s) [noise=true , address=145.79.227.11:4729 , piece_id=OVDGHBB33RRLVZ3ONAJP5GICGP6PB3FVWX7JUJQFQJN5OJ4TMGEA, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:48.316665091 (570h26m48.187748136s) [noise=true , address=77.90.31.10:4517 , piece_id=DIAMXFJZYC3BJIRBLAC4EAJ5WXCIBDG557PDLPK5DQGHMKCMFF7A, seg_pos={0 0}, upload=500121]: Uploading piece... Dec 29 09:55:50.005239292 (570h26m49.876322337s) [upload=500121]: Next returned nil segment Dec 29 09:55:50.005291762 (570h26m49.876374797s) [upload=500121]: Waiting for error group managing segments... Dec 29 09:55:50.298555816 (570h26m50.169638861s) [noise=true , address=145.79.227.39:4013 , piece_id=TPSFBRML57UXRNPBVWSC3VCXVFSKOUZHCJRVPBZK3SZ2HHXZVMDA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.306649337 (570h26m50.177732372s) [noise=true , address=145.79.227.41:4087 , piece_id=QHFWEZBDIACT53C7I37R55RXI3Y2ZDGR56P7N7B5KMVD6KRKIWEQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.318850028 (570h26m50.189933073s) [noise=true , address=145.79.227.38:4978 , piece_id=OZH3XZGQW52OFES45MJYVOKMNFAVULIPXGECYADHWD4HDPNOC5CQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.322765935 (570h26m50.19384898s) [noise=true , address=145.79.227.11:4729 , piece_id=OVDGHBB33RRLVZ3ONAJP5GICGP6PB3FVWX7JUJQFQJN5OJ4TMGEA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.324301569 (570h26m50.195384604s) [noise=true , address=145.79.227.6:4258 , piece_id=75ZAI4PQXC4ALQ6XR3DQAFJ3ESZIQCO6GQ6EAZNEU6TSOC7GQKGQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.496708931 (570h26m50.367791966s) [noise=true , address=77.90.31.10:4517 , piece_id=DIAMXFJZYC3BJIRBLAC4EAJ5WXCIBDG557PDLPK5DQGHMKCMFF7A, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.497362305 (570h26m50.36844534s) [noise=true , address=77.90.31.11:4555 , piece_id=ARLBME6S2RPXORLKMR3G4LE3NFTEISO3Y6OWM3SBUICX5322GWKQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.583064827 (570h26m50.454147872s) [noise=true , address=77.90.31.19:4844 , piece_id=MOUMHFHLLIVLXP5VCHZDVOCQRVVLAJE3IZIQQZQQDUVRAVVHIRBQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.626062862 (570h26m50.497145897s) [noise=true , address=77.90.31.13:4626 , piece_id=KRGHAN2HYDWV5PLEH4T66G5PT6BEZZFOFUHXPYSOUWCZHNLFZLFA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.632464392 (570h26m50.503547437s) [noise=true , address=45.91.116.13:4153 , piece_id=4DVDHFYX4BVL7JTFIRGAU5EFWNBFWQE6YJ3N55VN4X5LZHH2WUJA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.703846945 (570h26m50.57492999s) [noise=true , address=45.91.116.14:4192 , piece_id=JX5MVZOVMEQKQUFL6KLW7YPIU4FWZ6UVBH7RP3YMNVHN3TMOMIJA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:50.734108439 (570h26m50.605191484s) [noise=true , address=45.91.116.11:4120 , piece_id=QNPCJXQUWA72KXMVRJMLTR3EWG7KTSWTINF43ZA5SDEUZ7VVRFUA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:51.427016850 (570h26m51.298099885s) [noise=true , address=45.91.116.8:4046 , piece_id=TFZT7E5HA44MFCJVPYEYZ2FFU4NAPTYJENDISYYZAEQRLFAFXLHA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: <nil> Dec 29 09:55:51.427039252 (570h26m51.298122287s) [seg_pos={0 0}, upload=500121]: Segment reached optimal threshold of 13 pieces. Dec 29 09:55:51.427267056 (570h26m51.298350122s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.427278969 (570h26m51.298362014s) [noise=true , address=145.79.227.40:4049 , piece_id=XSBHW7ROQXSODFU4YTFF7576DYW2IQC5BJAFKBMBR3RYRFQQQFOQ, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.428909343 (570h26m51.299992389s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.428920104 (570h26m51.300003149s) [noise=true , address=145.79.227.49:4339 , piece_id=5HDMJ6KWCZ4JT34PZWKTIFDQ35KPF4YCYHK6ABF3DX2GOKI5RMQA, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.429082433 (570h26m51.300165478s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 09:55:51.429090428 (570h26m51.300173473s) [noise=true , address=45.91.116.6:4973 , piece_id=RS6Q33GNEW7DJZVHLF36W7HRIKFVUW4QQAWMCDEAM6DFAZKUZXHA, seg_pos={0 0}, upload=500121]: Long tail cancellation due to optimal threshold Dec 29 09:55:51.429113903 (570h26m51.300196968s) [seg_pos={0 0}, upload=500121]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:55:51.429194486 (570h26m51.300277531s) [upload=500121]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:55:51.429201420 (570h26m51.300284455s) [upload=500121]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:55:51.717085218 (570h26m51.588168253s) [upload=500121]: Done uploading segments... Dec 29 09:55:51.717101930 (570h26m51.588184975s) [upload=500121]: Upload finished. err: <nil> Duration: 30.003677931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.....................X.........] duration: 43.393560773s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:55:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 43.393586694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X.......X....................................XXX..XXX....] duration: 3.496546626s trace: https://jaeger.dev.storj.io/trace/742624afa397e34c FAIL (2025-12-29 09:55:16 UTC) orbiter warning: context deadline exceeded Duration: 3.496573559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X...] duration: 5.92300973s trace: https://jaeger.dev.storj.io/trace/5a7cbbb45c4b4b8c FAIL (2025-12-29 09:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ) context deadline exceeded ======LOGS====== (570h23m52.232832072s) [upload=500084]: Done starting upload Dec 29 09:52:52.361824590 (570h23m52.232907635s) [upload=500084]: Uploading segments... Dec 29 09:52:52.362017687 (570h23m52.233100723s) [upload=500084]: Got next segment. Inline: false Dec 29 09:52:52.362058715 (570h23m52.23314176s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:52:52.362063164 (570h23m52.233146209s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:52:52.612519413 (570h23m52.483602448s) [seg_pos={0 0}, upload=500084]: Begin upload segment... Dec 29 09:52:52.612563507 (570h23m52.483646542s) [seg_pos={0 0}, upload=500084]: Done begin upload segment. Dec 29 09:52:52.612590318 (570h23m52.483673353s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.612632949 (570h23m52.483715984s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613156135 (570h23m52.48423917s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613443091 (570h23m52.484526126s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613694040 (570h23m52.484777075s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614026432 (570h23m52.485109467s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614313879 (570h23m52.485396914s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614698993 (570h23m52.485782028s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.615026477 (570h23m52.486109512s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.616693621 (570h23m52.487776666s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.617755683 (570h23m52.488838728s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.619279004 (570h23m52.490362049s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.623236220 (570h23m52.494319255s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.624867486 (570h23m52.495950521s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.625280712 (570h23m52.496363747s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.626689435 (570h23m52.49777247s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:54.324818220 (570h23m54.195901255s) [upload=500084]: Next returned nil segment Dec 29 09:52:54.324837016 (570h23m54.195920051s) [upload=500084]: Waiting for error group managing segments... Dec 29 09:52:54.601169579 (570h23m54.472252614s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619198549 (570h23m54.490281594s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619857874 (570h23m54.490940919s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.621489801 (570h23m54.492572846s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.622527768 (570h23m54.493610803s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.644227437 (570h23m54.515310472s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.656138728 (570h23m54.527221763s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.864168611 (570h23m54.735251656s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.874282540 (570h23m54.745365575s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.889705578 (570h23m54.760788623s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.890083488 (570h23m54.761166533s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.898260057 (570h23m54.769343092s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973494456 (570h23m54.844577501s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973514615 (570h23m54.84459765s) [seg_pos={0 0}, upload=500084]: Segment reached optimal threshold of 13 pieces. Dec 29 09:52:54.973794317 (570h23m54.844877352s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:37944->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 09:52:54.973840014 (570h23m54.844923049s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973933442 (570h23m54.845016477s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 09:52:54.973953490 (570h23m54.845036525s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973959292 (570h23m54.845042327s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection Dec 29 09:52:54.973965133 (570h23m54.845048188s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.974033883 (570h23m54.845116929s) [seg_pos={0 0}, upload=500084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:52:54.974098667 (570h23m54.845181702s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:52:54.974103115 (570h23m54.84518615s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:52:55.274606061 (570h23m55.145689106s) [upload=500084]: Done uploading segments... Dec 29 09:52:55.274632622 (570h23m55.145715667s) [upload=500084]: Upload finished. err: <nil> Duration: 5.923033045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [...XX..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X..] duration: 6.00245101s trace: https://jaeger.dev.storj.io/trace/5a7cbbb45c4b4b8c FAIL (2025-12-29 09:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ) context deadline exceeded ======LOGS====== (570h23m52.232832072s) [upload=500084]: Done starting upload Dec 29 09:52:52.361824590 (570h23m52.232907635s) [upload=500084]: Uploading segments... Dec 29 09:52:52.362017687 (570h23m52.233100723s) [upload=500084]: Got next segment. Inline: false Dec 29 09:52:52.362058715 (570h23m52.23314176s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:52:52.362063164 (570h23m52.233146209s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:52:52.612519413 (570h23m52.483602448s) [seg_pos={0 0}, upload=500084]: Begin upload segment... Dec 29 09:52:52.612563507 (570h23m52.483646542s) [seg_pos={0 0}, upload=500084]: Done begin upload segment. Dec 29 09:52:52.612590318 (570h23m52.483673353s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.612632949 (570h23m52.483715984s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613156135 (570h23m52.48423917s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613443091 (570h23m52.484526126s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613694040 (570h23m52.484777075s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614026432 (570h23m52.485109467s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614313879 (570h23m52.485396914s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614698993 (570h23m52.485782028s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.615026477 (570h23m52.486109512s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.616693621 (570h23m52.487776666s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.617755683 (570h23m52.488838728s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.619279004 (570h23m52.490362049s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.623236220 (570h23m52.494319255s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.624867486 (570h23m52.495950521s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.625280712 (570h23m52.496363747s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.626689435 (570h23m52.49777247s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:54.324818220 (570h23m54.195901255s) [upload=500084]: Next returned nil segment Dec 29 09:52:54.324837016 (570h23m54.195920051s) [upload=500084]: Waiting for error group managing segments... Dec 29 09:52:54.601169579 (570h23m54.472252614s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619198549 (570h23m54.490281594s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619857874 (570h23m54.490940919s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.621489801 (570h23m54.492572846s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.622527768 (570h23m54.493610803s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.644227437 (570h23m54.515310472s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.656138728 (570h23m54.527221763s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.864168611 (570h23m54.735251656s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.874282540 (570h23m54.745365575s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.889705578 (570h23m54.760788623s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.890083488 (570h23m54.761166533s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.898260057 (570h23m54.769343092s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973494456 (570h23m54.844577501s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973514615 (570h23m54.84459765s) [seg_pos={0 0}, upload=500084]: Segment reached optimal threshold of 13 pieces. Dec 29 09:52:54.973794317 (570h23m54.844877352s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:37944->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 09:52:54.973840014 (570h23m54.844923049s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973933442 (570h23m54.845016477s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 09:52:54.973953490 (570h23m54.845036525s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973959292 (570h23m54.845042327s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection Dec 29 09:52:54.973965133 (570h23m54.845048188s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.974033883 (570h23m54.845116929s) [seg_pos={0 0}, upload=500084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:52:54.974098667 (570h23m54.845181702s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:52:54.974103115 (570h23m54.84518615s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:52:55.274606061 (570h23m55.145689106s) [upload=500084]: Done uploading segments... Dec 29 09:52:55.274632622 (570h23m55.145715667s) [upload=500084]: Upload finished. err: <nil> Duration: 6.002472302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.....................X........] duration: 42.073197553s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:54:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 42.073222049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X.......X....................................XXX..XXX...] duration: 5.001567594s trace: https://jaeger.dev.storj.io/trace/742624afa397e34c FAIL (2025-12-29 09:54:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001596168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....XX..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X.] duration: 17.112122711s trace: https://jaeger.dev.storj.io/trace/5a7cbbb45c4b4b8c FAIL (2025-12-29 09:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ) context deadline exceeded ======LOGS====== (570h23m52.232832072s) [upload=500084]: Done starting upload Dec 29 09:52:52.361824590 (570h23m52.232907635s) [upload=500084]: Uploading segments... Dec 29 09:52:52.362017687 (570h23m52.233100723s) [upload=500084]: Got next segment. Inline: false Dec 29 09:52:52.362058715 (570h23m52.23314176s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:52:52.362063164 (570h23m52.233146209s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:52:52.612519413 (570h23m52.483602448s) [seg_pos={0 0}, upload=500084]: Begin upload segment... Dec 29 09:52:52.612563507 (570h23m52.483646542s) [seg_pos={0 0}, upload=500084]: Done begin upload segment. Dec 29 09:52:52.612590318 (570h23m52.483673353s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.612632949 (570h23m52.483715984s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613156135 (570h23m52.48423917s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613443091 (570h23m52.484526126s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613694040 (570h23m52.484777075s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614026432 (570h23m52.485109467s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614313879 (570h23m52.485396914s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614698993 (570h23m52.485782028s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.615026477 (570h23m52.486109512s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.616693621 (570h23m52.487776666s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.617755683 (570h23m52.488838728s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.619279004 (570h23m52.490362049s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.623236220 (570h23m52.494319255s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.624867486 (570h23m52.495950521s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.625280712 (570h23m52.496363747s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.626689435 (570h23m52.49777247s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:54.324818220 (570h23m54.195901255s) [upload=500084]: Next returned nil segment Dec 29 09:52:54.324837016 (570h23m54.195920051s) [upload=500084]: Waiting for error group managing segments... Dec 29 09:52:54.601169579 (570h23m54.472252614s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619198549 (570h23m54.490281594s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619857874 (570h23m54.490940919s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.621489801 (570h23m54.492572846s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.622527768 (570h23m54.493610803s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.644227437 (570h23m54.515310472s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.656138728 (570h23m54.527221763s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.864168611 (570h23m54.735251656s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.874282540 (570h23m54.745365575s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.889705578 (570h23m54.760788623s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.890083488 (570h23m54.761166533s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.898260057 (570h23m54.769343092s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973494456 (570h23m54.844577501s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973514615 (570h23m54.84459765s) [seg_pos={0 0}, upload=500084]: Segment reached optimal threshold of 13 pieces. Dec 29 09:52:54.973794317 (570h23m54.844877352s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:37944->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 09:52:54.973840014 (570h23m54.844923049s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973933442 (570h23m54.845016477s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 09:52:54.973953490 (570h23m54.845036525s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973959292 (570h23m54.845042327s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection Dec 29 09:52:54.973965133 (570h23m54.845048188s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.974033883 (570h23m54.845116929s) [seg_pos={0 0}, upload=500084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:52:54.974098667 (570h23m54.845181702s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:52:54.974103115 (570h23m54.84518615s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:52:55.274606061 (570h23m55.145689106s) [upload=500084]: Done uploading segments... Dec 29 09:52:55.274632622 (570h23m55.145715667s) [upload=500084]: Upload finished. err: <nil> Duration: 17.11215332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.....................X.......] duration: 21.988884714s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:53:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 21.988907507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X.......X....................................XXX..XXX..] duration: 3.524992533s trace: https://jaeger.dev.storj.io/trace/742624afa397e34c FAIL (2025-12-29 09:53:37 UTC) orbiter warning: context deadline exceeded Duration: 3.525019514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....XX..XX....X.XX...XX...X.X.X...X.X.X.X..........XXXX.X.X] duration: 30.002463355s trace: https://jaeger.dev.storj.io/trace/5a7cbbb45c4b4b8c FAIL (2025-12-29 09:53:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ) context deadline exceeded ======LOGS====== (570h23m52.232832072s) [upload=500084]: Done starting upload Dec 29 09:52:52.361824590 (570h23m52.232907635s) [upload=500084]: Uploading segments... Dec 29 09:52:52.362017687 (570h23m52.233100723s) [upload=500084]: Got next segment. Inline: false Dec 29 09:52:52.362058715 (570h23m52.23314176s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 09:52:52.362063164 (570h23m52.233146209s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 09:52:52.612519413 (570h23m52.483602448s) [seg_pos={0 0}, upload=500084]: Begin upload segment... Dec 29 09:52:52.612563507 (570h23m52.483646542s) [seg_pos={0 0}, upload=500084]: Done begin upload segment. Dec 29 09:52:52.612590318 (570h23m52.483673353s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.612632949 (570h23m52.483715984s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613156135 (570h23m52.48423917s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613443091 (570h23m52.484526126s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.613694040 (570h23m52.484777075s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614026432 (570h23m52.485109467s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614313879 (570h23m52.485396914s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.614698993 (570h23m52.485782028s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.615026477 (570h23m52.486109512s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.616693621 (570h23m52.487776666s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.617755683 (570h23m52.488838728s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.619279004 (570h23m52.490362049s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.623236220 (570h23m52.494319255s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.624867486 (570h23m52.495950521s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.625280712 (570h23m52.496363747s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:52.626689435 (570h23m52.49777247s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Uploading piece... Dec 29 09:52:54.324818220 (570h23m54.195901255s) [upload=500084]: Next returned nil segment Dec 29 09:52:54.324837016 (570h23m54.195920051s) [upload=500084]: Waiting for error group managing segments... Dec 29 09:52:54.601169579 (570h23m54.472252614s) [noise=true , address=145.79.227.47:4267 , piece_id=FQNUCDPLKURM773CGXXFK73ECDJ7IMYEF427NH7C4TJNEBSRCVKQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619198549 (570h23m54.490281594s) [noise=true , address=145.79.227.5:4223 , piece_id=BIATD7DZFPN2RGQDUOMESWOF3LVDHOIRLTYPMOFN67CEXVVSI65Q, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.619857874 (570h23m54.490940919s) [noise=true , address=145.79.227.15:4874 , piece_id=GDFRCOODTOVDFLQCIGP7E5C5WJ37MP725252UKJA44DECQIDX3WA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.621489801 (570h23m54.492572846s) [noise=true , address=145.79.227.39:4013 , piece_id=WXALLLKE3K76Y5JKJLVKP7QW3RISFBF47XM5K62K4JFYFDFW4XZA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.622527768 (570h23m54.493610803s) [noise=true , address=145.79.227.50:4373 , piece_id=EQW52NOH5OH7PIP6VJLWXUTZRIADTZHXBUS764TAPATAY4GWAEZQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.644227437 (570h23m54.515310472s) [noise=true , address=145.79.227.41:4086 , piece_id=RG72VU7WN3LHS5GV72SPRA3LIJML7OZZ27U55GIUYGF3WUTT6BEA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.656138728 (570h23m54.527221763s) [noise=true , address=145.79.227.3:4149 , piece_id=K5P7JNYL5LCUSLTDNLQSEBXFT2SZC4IACC4QZXIBV5ILBIU3Q6IQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.864168611 (570h23m54.735251656s) [noise=true , address=45.91.116.7:4010 , piece_id=WXEFRYM7ETP7NJ43MGVX5XFQXZ3SN4NV3QNJ22GSMJRYEUG5TFWQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.874282540 (570h23m54.745365575s) [noise=true , address=77.90.31.15:4697 , piece_id=KC7ZUYHS2BAD6HPK7DJHOVUUPMVALGZIWCOFYYOOHXPFGHQXGXIA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.889705578 (570h23m54.760788623s) [noise=true , address=77.90.31.10:4517 , piece_id=PC5U3ZXQMHDSUA7SWXPX3TCGAFM7XDYYXLCCW56ANGLIORRNNJHA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.890083488 (570h23m54.761166533s) [noise=true , address=45.91.116.11:4118 , piece_id=L5JSTJO7WNZ54R7DWOMSIUOEKJCXZSSO6IBY47LKSLQQ7SWUJMVA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.898260057 (570h23m54.769343092s) [noise=true , address=77.90.31.17:4773 , piece_id=R7D5WEJENGRRBY3PTDHV25ON4LPRITJZ454OWAH4677IBQ3ATT4A, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973494456 (570h23m54.844577501s) [noise=true , address=45.91.116.14:4192 , piece_id=6DRYONW3Y42V2VKOMTHEK6XORJ36EAAA2ELIPSOYC4SHWC6QRGFQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: <nil> Dec 29 09:52:54.973514615 (570h23m54.84459765s) [seg_pos={0 0}, upload=500084]: Segment reached optimal threshold of 13 pieces. Dec 29 09:52:54.973794317 (570h23m54.844877352s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:37944->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 09:52:54.973840014 (570h23m54.844923049s) [noise=true , address=45.91.116.13:4154 , piece_id=2GT6UYRSFVBXHKQTANTOD4DB5QUCSU36BVL4ELWGL6365MX3YROA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973933442 (570h23m54.845016477s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 29 09:52:54.973953490 (570h23m54.845036525s) [noise=true , address=45.91.116.10:4083 , piece_id=NYG3YJOPERPCMXZWHHH5NUS5X5FQW4BOQVLZDPVGPZADGIJZTHCA, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.973959292 (570h23m54.845042327s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50816->45.91.116.8:4048: use of closed network connection Dec 29 09:52:54.973965133 (570h23m54.845048188s) [noise=true , address=45.91.116.8:4048 , piece_id=ZWCBTQQOZ2ZRKJTE5VAPANVYAT5KPM7BFEUZQMTX5O2OG2W5CMXQ, seg_pos={0 0}, upload=500084]: Long tail cancellation due to optimal threshold Dec 29 09:52:54.974033883 (570h23m54.845116929s) [seg_pos={0 0}, upload=500084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 09:52:54.974098667 (570h23m54.845181702s) [upload=500084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:52:54.974103115 (570h23m54.84518615s) [upload=500084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:52:55.274606061 (570h23m55.145689106s) [upload=500084]: Done uploading segments... Dec 29 09:52:55.274632622 (570h23m55.145715667s) [upload=500084]: Upload finished. err: <nil> Duration: 30.002485056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.....................X......] duration: 44.957160766s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:53:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 44.957175664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.....................X.....] duration: 46.701414626s trace: https://jaeger.dev.storj.io/trace/25e0f46b6f1b489 FAIL (2025-12-29 09:52:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA) context deadline exceeded; piecestore: (Node ID: 12tQRqowH4aDr9h3s1WLs1o7na1ueiafwgUeeHfAB6g7mcE6e1t, Piece ID: XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q) context deadline exceeded; piecestore: (Node ID: 17VVqv2A8JgKbRYuqxsdnPhomFxdBaznCHMDV21XFsCpuAFBNx, Piece ID: VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A) context deadline exceeded ======LOGS====== ec 29 09:48:03.863982713 (570h19m3.735065748s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864398334 (570h19m3.735481379s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.864692274 (570h19m3.735775319s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865058912 (570h19m3.736141957s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865259304 (570h19m3.736342349s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865389161 (570h19m3.736472196s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865717907 (570h19m3.736800982s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.865964416 (570h19m3.737047461s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.866642557 (570h19m3.737725592s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867148501 (570h19m3.738231546s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.867648493 (570h19m3.738731528s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868141632 (570h19m3.739224677s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.868734721 (570h19m3.739817756s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869296602 (570h19m3.740379647s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.869723254 (570h19m3.740806299s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:03.870864947 (570h19m3.741947992s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Uploading piece... Dec 29 09:48:05.167838121 (570h19m5.038921156s) [upload=500009]: Next returned nil segment Dec 29 09:48:05.167852067 (570h19m5.038935102s) [upload=500009]: Waiting for error group managing segments... Dec 29 09:48:05.527147153 (570h19m5.398230198s) [noise=true , address=64.49.13.154:4934 , piece_id=QYYM3I763C22IS2RPMCXH444Y3EO2EDJ5NV6AHUUGZ6JTHXDASCA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.590577975 (570h19m5.46166101s) [noise=true , address=64.49.13.139:4469 , piece_id=DAHCHYN3C77KA6H6VZRM2OYXP23BNRZAPG2GJ3OVWUKYTY6YV7JA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.715793997 (570h19m5.586877032s) [noise=true , address=145.79.227.11:4747 , piece_id=XODHSLEPODSOXBMCVSERUE2L7M4HY7DKPZM5GGXAS7BOQVXCNZIA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.735039442 (570h19m5.606122488s) [noise=true , address=145.79.227.46:4624 , piece_id=46XSIOBXDSENASRWCAWCV5QI5F6PZD2NRE4HDGIXH3UMQK5TRANA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.764843935 (570h19m5.63592697s) [noise=true , address=145.79.227.5:4246 , piece_id=6XV2ZFKEV3ANNF4UJKGFIBB2UTDIQ33R2HWYZE43YBN5TGF2BCXA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.770042754 (570h19m5.641125799s) [noise=true , address=145.79.227.33:4889 , piece_id=KSLHISPAMOBYSCWSCTQWZ3HVACLYYM2TIGYCYGZOIV4GMYOFHU7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.824052578 (570h19m5.695135623s) [noise=true , address=64.49.13.27:4957 , piece_id=M6I43TMCFC4OMSD7FZLRC5UYYUEQU2A4ZJVUIJL6YEB7VOH63HNA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.938632065 (570h19m5.80971511s) [noise=true , address=77.90.31.11:4582 , piece_id=VS3OODYLCZOGERAO7PQDNRDGRHJUJT5DGBPNB5YEH2Z5KE3PRS7A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:05.999798586 (570h19m5.870881631s) [noise=true , address=45.91.116.14:4213 , piece_id=XMLZNHU7B2WXDAMJ7BLP2FBUAGEVGUZJ775QLMUHSFZPYQMP6U5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.008094824 (570h19m5.879177859s) [noise=true , address=77.90.31.14:4665 , piece_id=JWQZYVQNQQPWTPIU3NP5NERKBHG23XM6TT6LGWGLZABJ5N3WM4SA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.016625949 (570h19m5.887708994s) [noise=true , address=77.90.31.16:4744 , piece_id=HWKQ6VUNLLD47QFD6H52GBAZ2GWEJYBBBK2SLPIIWZ4JFUFYD4GA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.355084778 (570h19m6.226167813s) [noise=true , address=193.149.164.143:4699 , piece_id=BKIUUJXLJX2D7NQMU3GU2VORVJSKAK72RMHTKACG6P7UZ6I2TC5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.613567090 (570h19m6.484650125s) [noise=true , address=145.79.227.44:4217 , piece_id=X5RVXFXM3KBMKBJLQ5V4ICHWJCIQXW3KQ4RBQTT3C4I3ZT6LYCHQ, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659018576 (570h19m6.530101621s) [noise=true , address=145.79.227.34:4923 , piece_id=VR6V7WNHEIZNGNZS2JLBXQIMD5DBGZU5QOMVPXMM252QXOT7JS3Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: <nil> Dec 29 09:48:06.659039256 (570h19m6.530122291s) [seg_pos={0 0}, upload=500009]: Segment reached optimal threshold of 14 pieces. Dec 29 09:48:06.659241020 (570h19m6.530324066s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38682->64.49.13.169:4707: use of closed network connection Dec 29 09:48:06.659248695 (570h19m6.53033174s) [noise=true , address=64.49.13.169:4707 , piece_id=ABOWWUNN3T2DCNH3OFDXQTNBAWVNWFU5GQF27CN2CNGPFI7ROA5Q, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659286196 (570h19m6.530369231s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1opDYSgBqBFTN6wZ5mrN5pJsnt9AH4y5sDr1yxE3Cjxg3vbq3v): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659296365 (570h19m6.53037941s) [noise=true , address=145.79.227.40:4054 , piece_id=DZWT3HCZL3CQTJ3OUNILYRKVSME722A2B5DQ34CICP2WJGO5D6AA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659467141 (570h19m6.530550186s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659473934 (570h19m6.530556979s) [noise=true , address=145.79.227.30:4770 , piece_id=LYZO2QIE7VOEJATFVKF4JBG4H4LB2QVCAB34KA5PIHMJRKNASJAA, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659514842 (570h19m6.530597887s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM): protocol: expected piece hash; context canceled; EOF Dec 29 09:48:06.659523388 (570h19m6.530606423s) [noise=true , address=77.90.31.19:4846 , piece_id=VLXXDOBAFN3H72UBUHOZMO2ONUPBWYTIRG665XMFE5N4OUYIYM4A, seg_pos={0 0}, upload=500009]: Long tail cancellation due to optimal threshold Dec 29 09:48:06.659546643 (570h19m6.530629688s) [seg_pos={0 0}, upload=500009]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 09:48:06.659625473 (570h19m6.530708508s) [upload=500009]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 09:48:06.659632456 (570h19m6.530715491s) [upload=500009]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 09:48:06.944761326 (570h19m6.815844381s) [upload=500009]: Done uploading segments... Dec 29 09:48:06.944786935 (570h19m6.81586997s) [upload=500009]: Upload finished. err: <nil> Duration: 46.701509948s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......X.......X....................................XXX..XXX.] duration: 5.001285646s trace: https://jaeger.dev.storj.io/trace/742624afa397e34c FAIL (2025-12-29 09:52:28 UTC) orbiter warning: context deadline exceeded Duration: 5.001305966s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)