Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [......X....X...........X.....XX.....X..X.X....X.XXXX....X.X.] duration: 3.399010693s trace: https://jaeger.dev.storj.io/trace/55132911f76a95a3 FAIL (2025-12-28 03:37:26 UTC) orbiter warning: context deadline exceeded Duration: 3.399035782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 38.435430857s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:36:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 38.435458891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XX] duration: 30.002367084s trace: https://jaeger.dev.storj.io/trace/31831a6e047f951a FAIL (2025-12-28 03:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HIQ5GYZGM6R7QLHEOTHNCUU5XYRD2AXVC2GMCR64FMBNSTC54ZWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: V2F45POPJZQEGD52K3ZJOFG7AX34SZPR3OYHQQUB3VDL3KJS455A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: KBK77ABFSGTE3PBWTFGM6FSTS26LJQGQA5B2VFRNBC6CHYRU67OQ) context deadline exceeded ======LOGS====== 75]: Starting upload Dec 28 03:36:15.726722036 (540h7m15.597805081s) [upload=473675]: Done starting upload Dec 28 03:36:15.726918680 (540h7m15.598001715s) [upload=473675]: Uploading segments... Dec 28 03:36:15.726953326 (540h7m15.598036361s) [upload=473675]: Got next segment. Inline: false Dec 28 03:36:15.726975057 (540h7m15.598058092s) [upload=473675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:36:15.726979516 (540h7m15.598062561s) [upload=473675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:36:15.973584063 (540h7m15.844667108s) [seg_pos={0 0}, upload=473675]: Begin upload segment... Dec 28 03:36:15.973643966 (540h7m15.844727001s) [seg_pos={0 0}, upload=473675]: Done begin upload segment. Dec 28 03:36:15.973682349 (540h7m15.844765394s) [noise=true , address=145.79.227.6:4257 , piece_id=7X2KVPMGZ63UBIDDD2SLAZSPLGPYBPSIG6BJIJQZYL6XWNXMDHSA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.973935181 (540h7m15.845018227s) [noise=true , address=77.90.31.18:4808 , piece_id=X3BEUITEQ3D73POHMFGBS3LYLFA5HET52BMM5XFW5SICYQVOP7NA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.975189079 (540h7m15.846272124s) [noise=true , address=145.79.227.19:4079 , piece_id=IYGFCXQH34XEHJC3JIR3LLJ6DKQG3B73NHEMFFKI5TZTCYK7CTWQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.975631862 (540h7m15.846714897s) [noise=true , address=145.79.227.17:4970 , piece_id=MOBEWRM5CBOJYXTJ5JQ6BZYBSCQM76O2Q7AXLU3WMOCYGSTIWSAQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.976074715 (540h7m15.84715776s) [noise=true , address=145.79.227.48:4301 , piece_id=ER2LXUHQBWHFLESYVZQTXI63QIJWDERBPDH2QJFLWD34E7FQ3UAQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.976472142 (540h7m15.847555187s) [noise=true , address=77.90.31.13:4629 , piece_id=FWP5HATNGHLFW5QXTX5GQVMSMBJFS52QN52P3CYZC4YGW6WQGJMA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.976851876 (540h7m15.847934911s) [noise=true , address=145.79.227.9:4365 , piece_id=34RM7GIPJW6SYOXKU5RDQYGRMNZKQ4PDSCFCVWABLT6Q4GC3EZAA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.978959629 (540h7m15.850042674s) [noise=true , address=145.79.227.11:4732 , piece_id=QT2UBMZNZZQQR2UVYOVFMK74V4D6VICQGCO6W75X2LCUDIHD4POQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.979464601 (540h7m15.850547646s) [noise=true , address=77.90.31.14:4663 , piece_id=J3WC5UHMCRDFM4CYE7SGBD3H3XNZWOIW5V5L5PPL3XWB46XC4OTA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.979914046 (540h7m15.850997081s) [noise=true , address=45.91.116.11:4117 , piece_id=HIQ5GYZGM6R7QLHEOTHNCUU5XYRD2AXVC2GMCR64FMBNSTC54ZWA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.980349485 (540h7m15.85143252s) [noise=true , address=77.90.31.17:4770 , piece_id=V2F45POPJZQEGD52K3ZJOFG7AX34SZPR3OYHQQUB3VDL3KJS455A, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.980786317 (540h7m15.851869362s) [noise=true , address=145.79.227.42:4122 , piece_id=EFXCJY7OGSHC5PGCZK5IJHHZYOJD75QU4AIL5TGHLGWOVMZEGEWA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.982523955 (540h7m15.85360699s) [noise=true , address=145.79.227.36:4519 , piece_id=SAMHAONOG46HNHDMT3KXEEBXT2XKRFJRBQGGFTO76S24O2CXK3RQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.983157823 (540h7m15.854240868s) [noise=true , address=145.79.227.12:4765 , piece_id=WY5XF5SGFV4M2NLOL4RQZUL77MPJ3N3RTRYGHMB6RLPSXZIZRECA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.983609323 (540h7m15.854692368s) [noise=true , address=145.79.227.29:4943 , piece_id=V23UUPDW2EE6AT3UB5P4ASUEDPEW74DHQS7QSXSDULCMEKHWSJNQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.984027389 (540h7m15.855110434s) [noise=true , address=77.90.31.10:4518 , piece_id=KBK77ABFSGTE3PBWTFGM6FSTS26LJQGQA5B2VFRNBC6CHYRU67OQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:17.699362663 (540h7m17.570445698s) [upload=473675]: Next returned nil segment Dec 28 03:36:17.699375518 (540h7m17.570458564s) [upload=473675]: Waiting for error group managing segments... Dec 28 03:36:17.982437926 (540h7m17.853520961s) [noise=true , address=145.79.227.17:4970 , piece_id=MOBEWRM5CBOJYXTJ5JQ6BZYBSCQM76O2Q7AXLU3WMOCYGSTIWSAQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:17.989240560 (540h7m17.860323595s) [noise=true , address=145.79.227.9:4365 , piece_id=34RM7GIPJW6SYOXKU5RDQYGRMNZKQ4PDSCFCVWABLT6Q4GC3EZAA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.013218588 (540h7m17.884301623s) [noise=true , address=145.79.227.42:4122 , piece_id=EFXCJY7OGSHC5PGCZK5IJHHZYOJD75QU4AIL5TGHLGWOVMZEGEWA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.044658043 (540h7m17.915741088s) [noise=true , address=145.79.227.29:4943 , piece_id=V23UUPDW2EE6AT3UB5P4ASUEDPEW74DHQS7QSXSDULCMEKHWSJNQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.231923248 (540h7m18.103006293s) [noise=true , address=45.91.116.11:4117 , piece_id=HIQ5GYZGM6R7QLHEOTHNCUU5XYRD2AXVC2GMCR64FMBNSTC54ZWA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.279888823 (540h7m18.150971858s) [noise=true , address=77.90.31.17:4770 , piece_id=V2F45POPJZQEGD52K3ZJOFG7AX34SZPR3OYHQQUB3VDL3KJS455A, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.293446518 (540h7m18.164529564s) [noise=true , address=77.90.31.10:4518 , piece_id=KBK77ABFSGTE3PBWTFGM6FSTS26LJQGQA5B2VFRNBC6CHYRU67OQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.297302371 (540h7m18.168385416s) [noise=true , address=77.90.31.13:4629 , piece_id=FWP5HATNGHLFW5QXTX5GQVMSMBJFS52QN52P3CYZC4YGW6WQGJMA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.604369287 (540h7m18.475452322s) [noise=true , address=145.79.227.12:4765 , piece_id=WY5XF5SGFV4M2NLOL4RQZUL77MPJ3N3RTRYGHMB6RLPSXZIZRECA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.806208162 (540h7m18.677291197s) [noise=true , address=145.79.227.6:4257 , piece_id=7X2KVPMGZ63UBIDDD2SLAZSPLGPYBPSIG6BJIJQZYL6XWNXMDHSA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.808390999 (540h7m18.679474044s) [noise=true , address=145.79.227.36:4519 , piece_id=SAMHAONOG46HNHDMT3KXEEBXT2XKRFJRBQGGFTO76S24O2CXK3RQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.905889933 (540h7m18.776972968s) [noise=true , address=145.79.227.11:4732 , piece_id=QT2UBMZNZZQQR2UVYOVFMK74V4D6VICQGCO6W75X2LCUDIHD4POQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.996573821 (540h7m18.867656856s) [noise=true , address=77.90.31.14:4663 , piece_id=J3WC5UHMCRDFM4CYE7SGBD3H3XNZWOIW5V5L5PPL3XWB46XC4OTA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.996599268 (540h7m18.867682313s) [seg_pos={0 0}, upload=473675]: Segment reached optimal threshold of 13 pieces. Dec 28 03:36:18.996829397 (540h7m18.867912452s) [noise=true , address=145.79.227.48:4301 , piece_id=ER2LXUHQBWHFLESYVZQTXI63QIJWDERBPDH2QJFLWD34E7FQ3UAQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:41842->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:41842->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->145.79.227.48:4301: use of closed network connection Dec 28 03:36:18.996839155 (540h7m18.8679222s) [noise=true , address=145.79.227.48:4301 , piece_id=ER2LXUHQBWHFLESYVZQTXI63QIJWDERBPDH2QJFLWD34E7FQ3UAQ, seg_pos={0 0}, upload=473675]: Long tail cancellation due to optimal threshold Dec 28 03:36:18.997311114 (540h7m18.868394159s) [noise=true , address=77.90.31.18:4808 , piece_id=X3BEUITEQ3D73POHMFGBS3LYLFA5HET52BMM5XFW5SICYQVOP7NA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:45910->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:36:18.997321614 (540h7m18.868404659s) [noise=true , address=77.90.31.18:4808 , piece_id=X3BEUITEQ3D73POHMFGBS3LYLFA5HET52BMM5XFW5SICYQVOP7NA, seg_pos={0 0}, upload=473675]: Long tail cancellation due to optimal threshold Dec 28 03:36:18.997474605 (540h7m18.86855765s) [noise=true , address=145.79.227.19:4079 , piece_id=IYGFCXQH34XEHJC3JIR3LLJ6DKQG3B73NHEMFFKI5TZTCYK7CTWQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 03:36:18.997483573 (540h7m18.868566618s) [noise=true , address=145.79.227.19:4079 , piece_id=IYGFCXQH34XEHJC3JIR3LLJ6DKQG3B73NHEMFFKI5TZTCYK7CTWQ, seg_pos={0 0}, upload=473675]: Long tail cancellation due to optimal threshold Dec 28 03:36:18.997503831 (540h7m18.868586866s) [seg_pos={0 0}, upload=473675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:36:18.997590977 (540h7m18.868674022s) [upload=473675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:36:18.997597049 (540h7m18.868680094s) [upload=473675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:36:19.286743141 (540h7m19.157826196s) [upload=473675]: Done uploading segments... Dec 28 03:36:19.286765954 (540h7m19.157848999s) [upload=473675]: Upload finished. err: <nil> Duration: 30.002380359s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.......X....X...........X.....XX.....X..X.X....X.XXXX....X.X] duration: 5.001289673s trace: https://jaeger.dev.storj.io/trace/55132911f76a95a3 FAIL (2025-12-28 03:36:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001318137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.X] duration: 30.003057154s trace: https://jaeger.dev.storj.io/trace/15791f08e0c45ccb FAIL (2025-12-28 03:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:35:41.560221920 (540h6m41.431304995s) [upload=473668]: Starting upload Dec 28 03:35:41.560290150 (540h6m41.431373205s) [upload=473668]: Done starting upload Dec 28 03:35:41.560408034 (540h6m41.431491079s) [upload=473668]: Uploading segments... Dec 28 03:35:41.560508826 (540h6m41.431591871s) [upload=473668]: Got next segment. Inline: false Dec 28 03:35:41.560540346 (540h6m41.431623391s) [upload=473668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:35:41.560544183 (540h6m41.431627218s) [upload=473668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:35:41.810109177 (540h6m41.681192222s) [seg_pos={0 0}, upload=473668]: Begin upload segment... Dec 28 03:35:41.810164402 (540h6m41.681247447s) [seg_pos={0 0}, upload=473668]: Done begin upload segment. Dec 28 03:35:41.810198207 (540h6m41.681281242s) [noise=true , address=145.79.227.50:4373 , piece_id=QWESKU633JUQF6TBTLGLZF3OAMD42ABQYNWI3GX6VMUIMZ2ABTAQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.810288559 (540h6m41.681371604s) [noise=true , address=45.91.116.13:4154 , piece_id=5LCW4SBCHQSVW4VIKAFSDDTXSFTXF5VMPA52C36XWMIMI5JXDEJQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.810954347 (540h6m41.682037392s) [noise=true , address=77.90.31.15:4700 , piece_id=FDNQEC5XRXFJAUVY2ASPXOKPICKMAQXYSFOE7TLJ7EFPY6IQIMOA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.811016385 (540h6m41.68209942s) [noise=true , address=145.79.227.5:4223 , piece_id=STQSOR4RKXJQIFZZQCSH3J4FPTS6CB4PNKGPRGTNM6I7BNSLHYRA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.811389505 (540h6m41.68247255s) [noise=true , address=45.91.116.10:4082 , piece_id=UZBLM5U6V7IJRT4YF7SRYWYGPMZZH3IWFEPECUYPXJZLTLLYKO5A, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.811682123 (540h6m41.682765158s) [noise=true , address=145.79.227.22:4511 , piece_id=KMHLWOXJTLWOYTHEM6HMKBVYRFK6DR7C7M47EGIWRBHDOWGE6UKA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812053189 (540h6m41.683136234s) [noise=true , address=77.90.31.21:4880 , piece_id=WA4AJZYRLSGQ6Y3H43ZK2WPZV7ANQOJ7F6ZD2X53UCVV4WHIEN5Q, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812086602 (540h6m41.683169647s) [noise=true , address=45.91.116.11:4118 , piece_id=GQLZAX5I7VZEVDXURQZZXLPDDUG3K47VOEKDV7N4Z2IT5FF5WAWQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812356096 (540h6m41.683439141s) [noise=true , address=77.90.31.14:4661 , piece_id=FRBSKDNXKZFK7PUBJR25NFZPERRMAZ2LYAU2UGLWCFV2KXMXG4OA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812584140 (540h6m41.683667176s) [noise=true , address=45.91.116.6:4973 , piece_id=2GCZS6MZKB27MR6RX66RVETYOYG4LFOUWJHKE7D2W6AYNHMLQXMQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812690653 (540h6m41.683773698s) [noise=true , address=145.79.227.10:4402 , piece_id=GR34SYSSX2EMFHOX67HAOYQFHSE4QDUC2NSEDUHWYB6S2SZWQ7MA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812963432 (540h6m41.684046467s) [noise=true , address=45.91.116.5:4299 , piece_id=QYZ7GOUIU6JS3ITOMFNKXP6ZKW6TOHKHIBCDQCEDIWCQ7O3256WA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.813077710 (540h6m41.684160745s) [noise=true , address=77.90.31.16:4735 , piece_id=3U5HCHJ4TSL6OW7K3RD7KM2TGURI33YI4XVC3YWYRLX4VNHBQRHA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.813564948 (540h6m41.684647993s) [noise=true , address=145.79.227.15:4875 , piece_id=UHMC5GLQR24KIPOLQALNDSCUV4DNAM2VDC7IYSLA4UCEDV6BXPJQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.814057345 (540h6m41.68514038s) [noise=true , address=145.79.227.31:4797 , piece_id=6R7XY2V6ZSFMTQNR3NRYQPS6AS6LBQSDYUHXJZTB6BJR6FZFIMNQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.815179121 (540h6m41.686262166s) [noise=true , address=145.79.227.16:4912 , piece_id=KIU63KH5T67OKWEJE45QZBNH5OZJIWPA6TT2BJFJAEWZ7AOTGKZA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:43.551664790 (540h6m43.422747825s) [upload=473668]: Next returned nil segment Dec 28 03:35:43.551678857 (540h6m43.422761893s) [upload=473668]: Waiting for error group managing segments... Dec 28 03:35:43.852598538 (540h6m43.723681573s) [noise=true , address=145.79.227.50:4373 , piece_id=QWESKU633JUQF6TBTLGLZF3OAMD42ABQYNWI3GX6VMUIMZ2ABTAQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.893047079 (540h6m43.764130114s) [noise=true , address=145.79.227.31:4797 , piece_id=6R7XY2V6ZSFMTQNR3NRYQPS6AS6LBQSDYUHXJZTB6BJR6FZFIMNQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.898665047 (540h6m43.769748092s) [noise=true , address=145.79.227.15:4875 , piece_id=UHMC5GLQR24KIPOLQALNDSCUV4DNAM2VDC7IYSLA4UCEDV6BXPJQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.918101117 (540h6m43.789184152s) [noise=true , address=145.79.227.16:4912 , piece_id=KIU63KH5T67OKWEJE45QZBNH5OZJIWPA6TT2BJFJAEWZ7AOTGKZA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.922833468 (540h6m43.793916503s) [noise=true , address=145.79.227.22:4511 , piece_id=KMHLWOXJTLWOYTHEM6HMKBVYRFK6DR7C7M47EGIWRBHDOWGE6UKA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.052738718 (540h6m43.923821753s) [noise=true , address=77.90.31.15:4700 , piece_id=FDNQEC5XRXFJAUVY2ASPXOKPICKMAQXYSFOE7TLJ7EFPY6IQIMOA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.121321501 (540h6m43.992404547s) [noise=true , address=77.90.31.16:4735 , piece_id=3U5HCHJ4TSL6OW7K3RD7KM2TGURI33YI4XVC3YWYRLX4VNHBQRHA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.129487191 (540h6m44.000570226s) [noise=true , address=77.90.31.14:4661 , piece_id=FRBSKDNXKZFK7PUBJR25NFZPERRMAZ2LYAU2UGLWCFV2KXMXG4OA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.149713595 (540h6m44.02079663s) [noise=true , address=45.91.116.11:4118 , piece_id=GQLZAX5I7VZEVDXURQZZXLPDDUG3K47VOEKDV7N4Z2IT5FF5WAWQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.176376094 (540h6m44.047459139s) [noise=true , address=45.91.116.5:4299 , piece_id=QYZ7GOUIU6JS3ITOMFNKXP6ZKW6TOHKHIBCDQCEDIWCQ7O3256WA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.185595660 (540h6m44.056678705s) [noise=true , address=77.90.31.21:4880 , piece_id=WA4AJZYRLSGQ6Y3H43ZK2WPZV7ANQOJ7F6ZD2X53UCVV4WHIEN5Q, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.413684970 (540h6m44.284768005s) [noise=true , address=45.91.116.10:4082 , piece_id=UZBLM5U6V7IJRT4YF7SRYWYGPMZZH3IWFEPECUYPXJZLTLLYKO5A, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.833075591 (540h6m44.704158626s) [noise=true , address=45.91.116.6:4973 , piece_id=2GCZS6MZKB27MR6RX66RVETYOYG4LFOUWJHKE7D2W6AYNHMLQXMQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.833094947 (540h6m44.704177992s) [seg_pos={0 0}, upload=473668]: Segment reached optimal threshold of 13 pieces. Dec 28 03:35:44.833270822 (540h6m44.704353857s) [noise=true , address=45.91.116.13:4154 , piece_id=5LCW4SBCHQSVW4VIKAFSDDTXSFTXF5VMPA52C36XWMIMI5JXDEJQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 03:35:44.833276392 (540h6m44.704359427s) [noise=true , address=45.91.116.13:4154 , piece_id=5LCW4SBCHQSVW4VIKAFSDDTXSFTXF5VMPA52C36XWMIMI5JXDEJQ, seg_pos={0 0}, upload=473668]: Long tail cancellation due to optimal threshold Dec 28 03:35:44.833468509 (540h6m44.704551544s) [noise=true , address=145.79.227.10:4402 , piece_id=GR34SYSSX2EMFHOX67HAOYQFHSE4QDUC2NSEDUHWYB6S2SZWQ7MA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 28 03:35:44.833477135 (540h6m44.70456018s) [noise=true , address=145.79.227.10:4402 , piece_id=GR34SYSSX2EMFHOX67HAOYQFHSE4QDUC2NSEDUHWYB6S2SZWQ7MA, seg_pos={0 0}, upload=473668]: Long tail cancellation due to optimal threshold Dec 28 03:35:44.833525548 (540h6m44.704608593s) [noise=true , address=145.79.227.5:4223 , piece_id=STQSOR4RKXJQIFZZQCSH3J4FPTS6CB4PNKGPRGTNM6I7BNSLHYRA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:34460->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:34460->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34460->145.79.227.5:4223: use of closed network connection Dec 28 03:35:44.833555123 (540h6m44.704638158s) [noise=true , address=145.79.227.5:4223 , piece_id=STQSOR4RKXJQIFZZQCSH3J4FPTS6CB4PNKGPRGTNM6I7BNSLHYRA, seg_pos={0 0}, upload=473668]: Long tail cancellation due to optimal threshold Dec 28 03:35:44.833636338 (540h6m44.704719373s) [seg_pos={0 0}, upload=473668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:35:44.833751537 (540h6m44.704834572s) [upload=473668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:35:44.833758581 (540h6m44.704841616s) [upload=473668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:35:45.127555028 (540h6m44.998638063s) [upload=473668]: Done uploading segments... Dec 28 03:35:45.127573944 (540h6m44.99865698s) [upload=473668]: Upload finished. err: <nil> Duration: 30.003080357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........X....X...........X.....XX.....X..X.X....X.XXXX....X.] duration: 4.506550443s trace: https://jaeger.dev.storj.io/trace/5cfee049c00fc4e6 FAIL (2025-12-28 03:35:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.506578438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 19.61304164s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:35:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 19.613060575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 49.987612304s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:34:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 49.987630518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.] duration: 6.028569024s trace: https://jaeger.dev.storj.io/trace/64419dc88841d4cb FAIL (2025-12-28 03:34:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ) context deadline exceeded ======LOGS====== Dec 28 03:33:45.747430799 (540h4m45.618513834s) [upload=473636]: Starting upload Dec 28 03:33:45.747491955 (540h4m45.618575s) [upload=473636]: Done starting upload Dec 28 03:33:45.747602135 (540h4m45.61868517s) [upload=473636]: Uploading segments... Dec 28 03:33:45.747766648 (540h4m45.618849683s) [upload=473636]: Got next segment. Inline: false Dec 28 03:33:45.747806584 (540h4m45.618889619s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:33:45.747810793 (540h4m45.618893838s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:33:45.997117805 (540h4m45.86820084s) [seg_pos={0 0}, upload=473636]: Begin upload segment... Dec 28 03:33:45.997173401 (540h4m45.868256436s) [seg_pos={0 0}, upload=473636]: Done begin upload segment. Dec 28 03:33:45.997214609 (540h4m45.868297644s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997299420 (540h4m45.868382455s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997729630 (540h4m45.868812665s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998073394 (540h4m45.869156429s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998402902 (540h4m45.869485937s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998942348 (540h4m45.870025393s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999343723 (540h4m45.870426758s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999755608 (540h4m45.870838653s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.000274987 (540h4m45.871358042s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.001711341 (540h4m45.872794407s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.003097301 (540h4m45.874180366s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.006293918 (540h4m45.877376953s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.007088891 (540h4m45.878171926s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.009599753 (540h4m45.880682788s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.013325828 (540h4m45.884408863s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.015511729 (540h4m45.886594774s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:47.734629898 (540h4m47.605712933s) [upload=473636]: Next returned nil segment Dec 28 03:33:47.734645909 (540h4m47.605728945s) [upload=473636]: Waiting for error group managing segments... Dec 28 03:33:48.020546311 (540h4m47.891629356s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035055670 (540h4m47.906138705s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035313530 (540h4m47.906396575s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.039157541 (540h4m47.910240576s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.089233874 (540h4m47.960316919s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.113787268 (540h4m47.984870303s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.287201077 (540h4m48.158284122s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.289709503 (540h4m48.160792538s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.347828651 (540h4m48.218911687s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.401863082 (540h4m48.272946127s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.720104827 (540h4m48.591187862s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.160352403 (540h4m49.031435438s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162353343 (540h4m49.033436389s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162365937 (540h4m49.033448982s) [seg_pos={0 0}, upload=473636]: Segment reached optimal threshold of 13 pieces. Dec 28 03:33:49.162672972 (540h4m49.033756007s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162682690 (540h4m49.033765735s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162781358 (540h4m49.033864413s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162790976 (540h4m49.033874011s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162857563 (540h4m49.033940598s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162864236 (540h4m49.033947271s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162882872 (540h4m49.033965917s) [seg_pos={0 0}, upload=473636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:33:49.162951192 (540h4m49.034034237s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:33:49.162957624 (540h4m49.034040669s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:33:49.519508356 (540h4m49.390591401s) [upload=473636]: Done uploading segments... Dec 28 03:33:49.519533103 (540h4m49.390616138s) [upload=473636]: Upload finished. err: <nil> Duration: 6.028596036s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X........X....X...........X.....XX.....X..X.X....X.XXXX....X] duration: 5.001557241s trace: https://jaeger.dev.storj.io/trace/5cfee049c00fc4e6 FAIL (2025-12-28 03:34:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001579525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX] duration: 30.002556657s trace: https://jaeger.dev.storj.io/trace/64419dc88841d4cb FAIL (2025-12-28 03:34:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ) context deadline exceeded ======LOGS====== Dec 28 03:33:45.747430799 (540h4m45.618513834s) [upload=473636]: Starting upload Dec 28 03:33:45.747491955 (540h4m45.618575s) [upload=473636]: Done starting upload Dec 28 03:33:45.747602135 (540h4m45.61868517s) [upload=473636]: Uploading segments... Dec 28 03:33:45.747766648 (540h4m45.618849683s) [upload=473636]: Got next segment. Inline: false Dec 28 03:33:45.747806584 (540h4m45.618889619s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:33:45.747810793 (540h4m45.618893838s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:33:45.997117805 (540h4m45.86820084s) [seg_pos={0 0}, upload=473636]: Begin upload segment... Dec 28 03:33:45.997173401 (540h4m45.868256436s) [seg_pos={0 0}, upload=473636]: Done begin upload segment. Dec 28 03:33:45.997214609 (540h4m45.868297644s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997299420 (540h4m45.868382455s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997729630 (540h4m45.868812665s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998073394 (540h4m45.869156429s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998402902 (540h4m45.869485937s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998942348 (540h4m45.870025393s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999343723 (540h4m45.870426758s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999755608 (540h4m45.870838653s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.000274987 (540h4m45.871358042s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.001711341 (540h4m45.872794407s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.003097301 (540h4m45.874180366s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.006293918 (540h4m45.877376953s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.007088891 (540h4m45.878171926s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.009599753 (540h4m45.880682788s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.013325828 (540h4m45.884408863s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.015511729 (540h4m45.886594774s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:47.734629898 (540h4m47.605712933s) [upload=473636]: Next returned nil segment Dec 28 03:33:47.734645909 (540h4m47.605728945s) [upload=473636]: Waiting for error group managing segments... Dec 28 03:33:48.020546311 (540h4m47.891629356s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035055670 (540h4m47.906138705s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035313530 (540h4m47.906396575s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.039157541 (540h4m47.910240576s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.089233874 (540h4m47.960316919s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.113787268 (540h4m47.984870303s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.287201077 (540h4m48.158284122s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.289709503 (540h4m48.160792538s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.347828651 (540h4m48.218911687s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.401863082 (540h4m48.272946127s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.720104827 (540h4m48.591187862s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.160352403 (540h4m49.031435438s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162353343 (540h4m49.033436389s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162365937 (540h4m49.033448982s) [seg_pos={0 0}, upload=473636]: Segment reached optimal threshold of 13 pieces. Dec 28 03:33:49.162672972 (540h4m49.033756007s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162682690 (540h4m49.033765735s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162781358 (540h4m49.033864413s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162790976 (540h4m49.033874011s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162857563 (540h4m49.033940598s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162864236 (540h4m49.033947271s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162882872 (540h4m49.033965917s) [seg_pos={0 0}, upload=473636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:33:49.162951192 (540h4m49.034034237s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:33:49.162957624 (540h4m49.034040669s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:33:49.519508356 (540h4m49.390591401s) [upload=473636]: Done uploading segments... Dec 28 03:33:49.519533103 (540h4m49.390616138s) [upload=473636]: Upload finished. err: <nil> Duration: 30.002579711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 56.309294596s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:34:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 56.309310476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXX] duration: 30.003677008s trace: https://jaeger.dev.storj.io/trace/ad79903e61ccdc5 FAIL (2025-12-28 03:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:32:45.432920286 (540h3m45.304003321s) [upload=473623]: Starting upload Dec 28 03:32:45.432988536 (540h3m45.304071571s) [upload=473623]: Done starting upload Dec 28 03:32:45.433054260 (540h3m45.304137295s) [upload=473623]: Uploading segments... Dec 28 03:32:45.433159150 (540h3m45.304242195s) [upload=473623]: Got next segment. Inline: false Dec 28 03:32:45.433207733 (540h3m45.304290778s) [upload=473623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:32:45.433212031 (540h3m45.304295066s) [upload=473623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:32:45.684023348 (540h3m45.555106393s) [seg_pos={0 0}, upload=473623]: Begin upload segment... Dec 28 03:32:45.684083673 (540h3m45.555166728s) [seg_pos={0 0}, upload=473623]: Done begin upload segment. Dec 28 03:32:45.684127787 (540h3m45.555210832s) [noise=true , address=145.79.227.50:4373 , piece_id=TOCN7YWQIHPOJGSA3ZKWNHQXO5LW7OVSIQ3ZLELL5QJKFCR3YM3Q, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.684218569 (540h3m45.555301604s) [noise=true , address=145.79.227.39:4014 , piece_id=2WPHA5HX2RFVYW7DXADOHHI6IXNO3IQC4WTQI32BGJNM4PA6LJ6A, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.684836176 (540h3m45.555919221s) [noise=true , address=145.79.227.3:4150 , piece_id=EHNKMGGIXZ6R3MWXFJJMYGE7Z3IMTFUUTV7DB7URIITI3YSFN7GA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.685309237 (540h3m45.556392282s) [noise=true , address=145.79.227.48:4303 , piece_id=WCGQ3G6LPKRNVGLFZOI6LRAKU2UCP4W2RVTW6OERM4UUBF43SXYA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.685787818 (540h3m45.556870863s) [noise=true , address=45.91.116.10:4083 , piece_id=C4CXR3ESUIIAN4GN3KKGEAWVNLML6BF5N63ST7ODJJR2MC7V6I5A, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.686294613 (540h3m45.557377658s) [noise=true , address=77.90.31.13:4626 , piece_id=QNRDQ7XH7AOT5736RYPUE4UKTABGEQYWPXENXFIRVKLLGSFAGBVA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.686820805 (540h3m45.55790385s) [noise=true , address=145.79.227.13:4803 , piece_id=52UFG7YBZKWRUZAQZ67JIKJRI264RD4U2J4N7LMMW4HSNSUAGLGQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.687243279 (540h3m45.558326314s) [noise=true , address=145.79.227.41:4087 , piece_id=VUUUFYEJQQ3TAOS6TUSYNWM4F52ASJKW6Z6M2MU6DCSNFXKV2CRA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.687709066 (540h3m45.558792112s) [noise=true , address=45.91.116.12:4227 , piece_id=J6ZKJWDCY3VU6O3UKFQJ52ZBF3LKD46D3P3XYRKJYC224D5D43AQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.689573326 (540h3m45.560656371s) [noise=true , address=145.79.227.34:4907 , piece_id=OQBOWOUHQJTE7RAPL5AYLHJQS5QFSZ6YHF5N2R6SKTV7SPREOZQQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.689987805 (540h3m45.56107084s) [noise=true , address=77.90.31.12:4589 , piece_id=UQKQEFGS5RXNWBGJZIVJBKYCBK53Q3FGAFN4KIAO3S3R2OIAWDXA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.691992142 (540h3m45.563075187s) [noise=true , address=77.90.31.19:4841 , piece_id=QWRQFEG2QCIUSCJR2BADBU45NS2CZQ55EYRAX2QFJWUF3NHGC4HQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.693969768 (540h3m45.565052813s) [noise=true , address=145.79.227.36:4518 , piece_id=CBYXXQCS6HKYC2RHHZEJDYGJIL5KBI76DGI4VNMLB2LEHS4V75QA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.696001777 (540h3m45.567084832s) [noise=true , address=145.79.227.12:4767 , piece_id=SLQPZHCS5FS4FXAM6N6INOZKOTZYD3F5EJ62PFZ6NR3CHAGORFRA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.697787257 (540h3m45.568870302s) [noise=true , address=45.91.116.8:4045 , piece_id=75B5J6C6NUZ55MWQDM3PTPUHFCLSTVMHFUS3FJWC2UWG6K3HACVQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.699666956 (540h3m45.570750001s) [noise=true , address=45.91.116.9:4263 , piece_id=W2OP4HIHE7LJ7KMKYA5SJU544WWKD6CYZ5L5XOC3CJ64ZQLSF6ZQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:47.470574951 (540h3m47.341657996s) [upload=473623]: Next returned nil segment Dec 28 03:32:47.470598706 (540h3m47.341681751s) [upload=473623]: Waiting for error group managing segments... Dec 28 03:32:47.767904355 (540h3m47.6389874s) [noise=true , address=145.79.227.41:4087 , piece_id=VUUUFYEJQQ3TAOS6TUSYNWM4F52ASJKW6Z6M2MU6DCSNFXKV2CRA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.775433041 (540h3m47.646516076s) [noise=true , address=145.79.227.3:4150 , piece_id=EHNKMGGIXZ6R3MWXFJJMYGE7Z3IMTFUUTV7DB7URIITI3YSFN7GA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.791343157 (540h3m47.662426222s) [noise=true , address=145.79.227.50:4373 , piece_id=TOCN7YWQIHPOJGSA3ZKWNHQXO5LW7OVSIQ3ZLELL5QJKFCR3YM3Q, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.903554436 (540h3m47.774637481s) [noise=true , address=77.90.31.19:4841 , piece_id=QWRQFEG2QCIUSCJR2BADBU45NS2CZQ55EYRAX2QFJWUF3NHGC4HQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.965550146 (540h3m47.836633191s) [noise=true , address=77.90.31.12:4589 , piece_id=UQKQEFGS5RXNWBGJZIVJBKYCBK53Q3FGAFN4KIAO3S3R2OIAWDXA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.034511110 (540h3m47.905594155s) [noise=true , address=45.91.116.8:4045 , piece_id=75B5J6C6NUZ55MWQDM3PTPUHFCLSTVMHFUS3FJWC2UWG6K3HACVQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.061535999 (540h3m47.932619034s) [noise=true , address=45.91.116.9:4263 , piece_id=W2OP4HIHE7LJ7KMKYA5SJU544WWKD6CYZ5L5XOC3CJ64ZQLSF6ZQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.285918931 (540h3m48.157001967s) [noise=true , address=145.79.227.48:4303 , piece_id=WCGQ3G6LPKRNVGLFZOI6LRAKU2UCP4W2RVTW6OERM4UUBF43SXYA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.321590296 (540h3m48.192673331s) [noise=true , address=145.79.227.13:4803 , piece_id=52UFG7YBZKWRUZAQZ67JIKJRI264RD4U2J4N7LMMW4HSNSUAGLGQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.324041594 (540h3m48.19512464s) [noise=true , address=145.79.227.36:4518 , piece_id=CBYXXQCS6HKYC2RHHZEJDYGJIL5KBI76DGI4VNMLB2LEHS4V75QA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.354092186 (540h3m48.225175221s) [noise=true , address=45.91.116.12:4227 , piece_id=J6ZKJWDCY3VU6O3UKFQJ52ZBF3LKD46D3P3XYRKJYC224D5D43AQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.560567850 (540h3m48.431650895s) [noise=true , address=77.90.31.13:4626 , piece_id=QNRDQ7XH7AOT5736RYPUE4UKTABGEQYWPXENXFIRVKLLGSFAGBVA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.861348917 (540h3m48.732431952s) [noise=true , address=145.79.227.39:4014 , piece_id=2WPHA5HX2RFVYW7DXADOHHI6IXNO3IQC4WTQI32BGJNM4PA6LJ6A, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.861369105 (540h3m48.73245214s) [seg_pos={0 0}, upload=473623]: Segment reached optimal threshold of 13 pieces. Dec 28 03:32:48.861606227 (540h3m48.732689262s) [noise=true , address=145.79.227.34:4907 , piece_id=OQBOWOUHQJTE7RAPL5AYLHJQS5QFSZ6YHF5N2R6SKTV7SPREOZQQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:48.861613130 (540h3m48.732696165s) [noise=true , address=145.79.227.34:4907 , piece_id=OQBOWOUHQJTE7RAPL5AYLHJQS5QFSZ6YHF5N2R6SKTV7SPREOZQQ, seg_pos={0 0}, upload=473623]: Long tail cancellation due to optimal threshold Dec 28 03:32:48.861987543 (540h3m48.733070588s) [noise=true , address=45.91.116.10:4083 , piece_id=C4CXR3ESUIIAN4GN3KKGEAWVNLML6BF5N63ST7ODJJR2MC7V6I5A, seg_pos={0 0}, upload=473623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:41156->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:32:48.862002913 (540h3m48.733085958s) [noise=true , address=45.91.116.10:4083 , piece_id=C4CXR3ESUIIAN4GN3KKGEAWVNLML6BF5N63ST7ODJJR2MC7V6I5A, seg_pos={0 0}, upload=473623]: Long tail cancellation due to optimal threshold Dec 28 03:32:48.862056544 (540h3m48.733139599s) [noise=true , address=145.79.227.12:4767 , piece_id=SLQPZHCS5FS4FXAM6N6INOZKOTZYD3F5EJ62PFZ6NR3CHAGORFRA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:48.862066213 (540h3m48.733149248s) [noise=true , address=145.79.227.12:4767 , piece_id=SLQPZHCS5FS4FXAM6N6INOZKOTZYD3F5EJ62PFZ6NR3CHAGORFRA, seg_pos={0 0}, upload=473623]: Long tail cancellation due to optimal threshold Dec 28 03:32:48.862086552 (540h3m48.733169587s) [seg_pos={0 0}, upload=473623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:32:48.862156365 (540h3m48.73323941s) [upload=473623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:32:48.862161033 (540h3m48.733244068s) [upload=473623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:32:49.172333874 (540h3m49.043416919s) [upload=473623]: Done uploading segments... Dec 28 03:32:49.172354133 (540h3m49.043437168s) [upload=473623]: Upload finished. err: <nil> Duration: 30.003702196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 24.330730747s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:33:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 24.330748532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X........X....X...........X.....XX.....X..X.X....X.XXXX....] duration: 4.864477707s trace: https://jaeger.dev.storj.io/trace/58e23bd7ac68184a FAIL (2025-12-28 03:33:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.86450639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 53.289843292s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:32:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 53.289862168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXX] duration: 30.004599396s trace: https://jaeger.dev.storj.io/trace/31ec26dc84c3e512 FAIL (2025-12-28 03:32:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 7M5CYZHNS7LFSHJ7FPXXNZBTEWRRCIWK2ZMDFKZLPGJRFKLRE4YA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: Q4RKNSG3EFMPT2ERPWEOAISOKQ2IKDQQHYHAHZMHTYCHP5MI4QRQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QMVRPGAB7U7AE5N3UABMOBK2BJJSAIYAPYVSLJLKKASKIXXJJSRQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 27AVXPXTMI7JL242QAZJ7FNCBH3J5HNGPHNAJ23HD5ZUXJUFLBQA) context deadline exceeded ======LOGS====== Dec 28 03:32:13.492523947 (540h3m13.363606982s) [upload=473617]: Starting upload Dec 28 03:32:13.492584753 (540h3m13.363667798s) [upload=473617]: Done starting upload Dec 28 03:32:13.492681097 (540h3m13.363764132s) [upload=473617]: Uploading segments... Dec 28 03:32:13.492784273 (540h3m13.363867309s) [upload=473617]: Got next segment. Inline: false Dec 28 03:32:13.492821013 (540h3m13.363904048s) [upload=473617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:32:13.492827175 (540h3m13.36391023s) [upload=473617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:32:13.737278110 (540h3m13.608361146s) [seg_pos={0 0}, upload=473617]: Begin upload segment... Dec 28 03:32:13.737331211 (540h3m13.608414246s) [seg_pos={0 0}, upload=473617]: Done begin upload segment. Dec 28 03:32:13.737423387 (540h3m13.608506432s) [noise=true , address=45.91.116.12:4226 , piece_id=A2XYJO3YSMGLEETT3B2EFATUSAQYETIU677ILBXVF3CQM7HMRITQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.737439878 (540h3m13.608522913s) [noise=true , address=77.90.31.19:4844 , piece_id=OGNDXD2GDQ4D2MTXWKLFNSSS6D2SJWQIKB4MTI2NF47RYB33MKOQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.738017999 (540h3m13.609101044s) [noise=true , address=145.79.227.13:4803 , piece_id=TGFZYOG6U46MLIJW5KNKVXA7RRETQQHQKNBVGUCL4EYAI4DHG3KQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.738369869 (540h3m13.609452904s) [noise=true , address=45.91.116.7:4012 , piece_id=QMVRPGAB7U7AE5N3UABMOBK2BJJSAIYAPYVSLJLKKASKIXXJJSRQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.738758579 (540h3m13.609841624s) [noise=true , address=145.79.227.52:4445 , piece_id=N3EQPQA4MNUNV343NBC326N3QNNAHOIO57BGLEXMFNYRJC7UAYEA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.739114177 (540h3m13.610197212s) [noise=true , address=45.91.116.14:4192 , piece_id=27AVXPXTMI7JL242QAZJ7FNCBH3J5HNGPHNAJ23HD5ZUXJUFLBQA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.739449465 (540h3m13.6105325s) [noise=true , address=145.79.227.17:4971 , piece_id=NIRGOZ4GZMFAO4RYI2LMU5LTI3PAZZR6OP7W7IWDINC6VZRXYFFQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.739861930 (540h3m13.610944975s) [noise=true , address=45.91.116.10:4082 , piece_id=EKPQZ3FD6HTXWK4WPCZ5RQLPCHLHPA7TJHCPQ4HE3XGVJIF3MMWQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.740696800 (540h3m13.611779835s) [noise=true , address=145.79.227.11:4732 , piece_id=CLTFD7G455ITSS3H5UE4W5C4JFE6SMARMUHNIW4VHU5IPCMNW2VA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.741769413 (540h3m13.612852458s) [noise=true , address=77.90.31.16:4735 , piece_id=7M5CYZHNS7LFSHJ7FPXXNZBTEWRRCIWK2ZMDFKZLPGJRFKLRE4YA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.742052101 (540h3m13.613135136s) [noise=true , address=145.79.227.15:4875 , piece_id=CXODGGZMXY6YBWX2R2KIP46WXICJO7VFWS3WOGPNRDBJS3Q4OKSQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.743410868 (540h3m13.614493903s) [noise=true , address=45.91.116.5:4297 , piece_id=Q4RKNSG3EFMPT2ERPWEOAISOKQ2IKDQQHYHAHZMHTYCHP5MI4QRQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.744673151 (540h3m13.615756187s) [noise=true , address=145.79.227.27:4691 , piece_id=QSZTQRPW4OEDJ7KBS3F5REBBGCHNXW3CKBCKLU635B3X45VOY2EQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.745903385 (540h3m13.61698643s) [noise=true , address=145.79.227.45:4229 , piece_id=UNJV7CQVHNDHXUZJDOE3GQ44EMXV425BYDYJYRFNM2IXYTW7E45A, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.746211340 (540h3m13.617294385s) [noise=true , address=77.90.31.13:4627 , piece_id=KTVTTKB7CLV266YSAVHD32GK2QM4PEFDWYLII5ZH5XJA4KKQXVVQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.750079256 (540h3m13.621162291s) [noise=true , address=145.79.227.50:4373 , piece_id=ZD4BLOXH7CZJRA2VBKS7HPEQDPMM4SD2G6UF24W25SEAJKXMI7ZQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:15.464465544 (540h3m15.335548579s) [upload=473617]: Next returned nil segment Dec 28 03:32:15.464493097 (540h3m15.335576142s) [upload=473617]: Waiting for error group managing segments... Dec 28 03:32:15.758258825 (540h3m15.62934187s) [noise=true , address=145.79.227.52:4445 , piece_id=N3EQPQA4MNUNV343NBC326N3QNNAHOIO57BGLEXMFNYRJC7UAYEA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.759138911 (540h3m15.630221946s) [noise=true , address=145.79.227.50:4373 , piece_id=ZD4BLOXH7CZJRA2VBKS7HPEQDPMM4SD2G6UF24W25SEAJKXMI7ZQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.766146685 (540h3m15.63722972s) [noise=true , address=145.79.227.13:4803 , piece_id=TGFZYOG6U46MLIJW5KNKVXA7RRETQQHQKNBVGUCL4EYAI4DHG3KQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.773461294 (540h3m15.644544339s) [noise=true , address=145.79.227.27:4691 , piece_id=QSZTQRPW4OEDJ7KBS3F5REBBGCHNXW3CKBCKLU635B3X45VOY2EQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.773573918 (540h3m15.644656953s) [noise=true , address=145.79.227.17:4971 , piece_id=NIRGOZ4GZMFAO4RYI2LMU5LTI3PAZZR6OP7W7IWDINC6VZRXYFFQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.918769555 (540h3m15.7898526s) [noise=true , address=77.90.31.16:4735 , piece_id=7M5CYZHNS7LFSHJ7FPXXNZBTEWRRCIWK2ZMDFKZLPGJRFKLRE4YA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.920111019 (540h3m15.791194054s) [noise=true , address=77.90.31.13:4627 , piece_id=KTVTTKB7CLV266YSAVHD32GK2QM4PEFDWYLII5ZH5XJA4KKQXVVQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.957983063 (540h3m15.829066098s) [noise=true , address=45.91.116.5:4297 , piece_id=Q4RKNSG3EFMPT2ERPWEOAISOKQ2IKDQQHYHAHZMHTYCHP5MI4QRQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.004341438 (540h3m15.875424483s) [noise=true , address=45.91.116.14:4192 , piece_id=27AVXPXTMI7JL242QAZJ7FNCBH3J5HNGPHNAJ23HD5ZUXJUFLBQA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.029985688 (540h3m15.901068733s) [noise=true , address=77.90.31.19:4844 , piece_id=OGNDXD2GDQ4D2MTXWKLFNSSS6D2SJWQIKB4MTI2NF47RYB33MKOQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.062195312 (540h3m15.933278347s) [noise=true , address=45.91.116.7:4012 , piece_id=QMVRPGAB7U7AE5N3UABMOBK2BJJSAIYAPYVSLJLKKASKIXXJJSRQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.091152611 (540h3m15.962235656s) [noise=true , address=45.91.116.12:4226 , piece_id=A2XYJO3YSMGLEETT3B2EFATUSAQYETIU677ILBXVF3CQM7HMRITQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.122380804 (540h3m15.99346384s) [noise=true , address=45.91.116.10:4082 , piece_id=EKPQZ3FD6HTXWK4WPCZ5RQLPCHLHPA7TJHCPQ4HE3XGVJIF3MMWQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.122399991 (540h3m15.993483027s) [seg_pos={0 0}, upload=473617]: Segment reached optimal threshold of 13 pieces. Dec 28 03:32:16.122643836 (540h3m15.993726871s) [noise=true , address=145.79.227.11:4732 , piece_id=CLTFD7G455ITSS3H5UE4W5C4JFE6SMARMUHNIW4VHU5IPCMNW2VA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:36216->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36216->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36216->145.79.227.11:4732: use of closed network connection Dec 28 03:32:16.122650258 (540h3m15.993733303s) [noise=true , address=145.79.227.11:4732 , piece_id=CLTFD7G455ITSS3H5UE4W5C4JFE6SMARMUHNIW4VHU5IPCMNW2VA, seg_pos={0 0}, upload=473617]: Long tail cancellation due to optimal threshold Dec 28 03:32:16.122942955 (540h3m15.99402599s) [noise=true , address=145.79.227.45:4229 , piece_id=UNJV7CQVHNDHXUZJDOE3GQ44EMXV425BYDYJYRFNM2IXYTW7E45A, seg_pos={0 0}, upload=473617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:16.122949969 (540h3m15.994033004s) [noise=true , address=145.79.227.45:4229 , piece_id=UNJV7CQVHNDHXUZJDOE3GQ44EMXV425BYDYJYRFNM2IXYTW7E45A, seg_pos={0 0}, upload=473617]: Long tail cancellation due to optimal threshold Dec 28 03:32:16.123062663 (540h3m15.994145708s) [noise=true , address=145.79.227.15:4875 , piece_id=CXODGGZMXY6YBWX2R2KIP46WXICJO7VFWS3WOGPNRDBJS3Q4OKSQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:16.123068584 (540h3m15.994151619s) [noise=true , address=145.79.227.15:4875 , piece_id=CXODGGZMXY6YBWX2R2KIP46WXICJO7VFWS3WOGPNRDBJS3Q4OKSQ, seg_pos={0 0}, upload=473617]: Long tail cancellation due to optimal threshold Dec 28 03:32:16.123086187 (540h3m15.994169232s) [seg_pos={0 0}, upload=473617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:32:16.123154668 (540h3m15.994237703s) [upload=473617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:32:16.123159989 (540h3m15.994243024s) [upload=473617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:32:16.410762552 (540h3m16.281845607s) [upload=473617]: Done uploading segments... Dec 28 03:32:16.410786316 (540h3m16.281869361s) [upload=473617]: Upload finished. err: <nil> Duration: 30.004621861s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..X........X....X...........X.....XX.....X..X.X....X.XXXX...] duration: 3.796909743s trace: https://jaeger.dev.storj.io/trace/58e23bd7ac68184a FAIL (2025-12-28 03:32:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.796937785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 59.092410877s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:31:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 59.092439793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XX] duration: 30.003413157s trace: https://jaeger.dev.storj.io/trace/587a0935156f8bb3 FAIL (2025-12-28 03:31:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:31:16.865111659 (540h2m16.736194704s) [upload=473602]: Starting upload Dec 28 03:31:16.865193415 (540h2m16.73627645s) [upload=473602]: Done starting upload Dec 28 03:31:16.865445766 (540h2m16.736528801s) [upload=473602]: Uploading segments... Dec 28 03:31:16.865542509 (540h2m16.736625544s) [upload=473602]: Got next segment. Inline: false Dec 28 03:31:16.865588938 (540h2m16.736671973s) [upload=473602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:31:16.865594178 (540h2m16.736677223s) [upload=473602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:31:17.132626322 (540h2m17.003709357s) [seg_pos={0 0}, upload=473602]: Begin upload segment... Dec 28 03:31:17.132674324 (540h2m17.003757359s) [seg_pos={0 0}, upload=473602]: Done begin upload segment. Dec 28 03:31:17.132713869 (540h2m17.003796904s) [noise=true , address=145.79.227.42:4122 , piece_id=UKJ5MZBVCY7N7EUVCNJRTTZ5HROSMKNFKPGKRONPKSL5OMLGJROQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.132722625 (540h2m17.00380567s) [noise=true , address=145.79.227.11:4729 , piece_id=VWLRLCD75L4CY7GWINKD4GNB2IBPAKRHIJ7ZJLOXOSBOT773UMCQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.132763813 (540h2m17.003846858s) [noise=true , address=145.79.227.46:4603 , piece_id=BJ5UIOOWRPSV625DSQIQOG5SAJVAHCJUMSSN2WMTPG7ROTK2AUVA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.133133057 (540h2m17.004216092s) [noise=true , address=77.90.31.21:4877 , piece_id=ETKK3JSU72QSYPAYNTNHQRXHFFYEJFA46V5YXZ5PFPFYOK6WQNGQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.133456452 (540h2m17.004539498s) [noise=true , address=45.91.116.5:4299 , piece_id=KOPWB3QKQL2YXKW63RENR5S6HUPHSTBIN2CVETQ2CMONY2AN5PFQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.133822780 (540h2m17.004905816s) [noise=true , address=145.79.227.36:4517 , piece_id=XNJGZJLNLXTMCA5BQ4MKD7M5XX2HTXRZ7LRMIAIXXZ5OTZOTEVMQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.134130917 (540h2m17.005213952s) [noise=true , address=145.79.227.41:4087 , piece_id=KD4LRWNGE67IN4FFMGEA2Z6MJSTDESZW5WSD5JLR6FRJWD4PKLXA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.135620503 (540h2m17.006703538s) [noise=true , address=45.91.116.14:4192 , piece_id=C5G7G6CQHJWIKJFE2GH76PKOKC73GM2A3ZXD5JEPNZB7SGRW7AQQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.136748932 (540h2m17.007831977s) [noise=true , address=77.90.31.12:4590 , piece_id=BDO6LJTEQNFNDRLZ3FBPH6CALSYC44JINIGZZPCYGKOEGEGJ5YUA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.137884895 (540h2m17.00896793s) [noise=true , address=145.79.227.38:4977 , piece_id=4R6P6GRVIFFDNKYAO6DYVRXL6LPQBZY42EAYY4SIIJTTYLNTKPPQ, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.139116410 (540h2m17.010199455s) [noise=true , address=77.90.31.10:4519 , piece_id=QCS44OTWILK6YLKZVW2XTUXXQPT3SP3CI4Z2DNVESJL4NNDFDYOA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.140306376 (540h2m17.011389411s) [noise=true , address=145.79.227.4:4187 , piece_id=5RWXHVIVRJ3ZED7WQLYYJNJOFX2SBEKBFH5WAGNWQFPO4EBXOJWA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.141366404 (540h2m17.012449449s) [noise=true , address=145.79.227.5:4222 , piece_id=AEB6HPEMVMNAVVJ4OBPGD37WVFEYFMVEZTWWN7YFKZ3HZSUT6D4Q, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.141927723 (540h2m17.013010758s) [noise=true , address=45.91.116.6:4974 , piece_id=W22IZ2D2BATH6E7YRVGU6C2ITD2N657F5JHGHSLV74TBYOVWGTYA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.143596260 (540h2m17.014679305s) [noise=true , address=145.79.227.33:4870 , piece_id=TLBT76FXM5I3I3VVJZDHNVMC2IU3Q57SBRXU6JHGBMPMCIEHTPOA, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:17.143821679 (540h2m17.014904714s) [noise=true , address=145.79.227.17:4971 , piece_id=FBNJBASX5WPSGN2ZWD5OO455FUR2GB4HEOINSXVH5KDQKX5J3Y4A, seg_pos={0 0}, upload=473602]: Uploading piece... Dec 28 03:31:18.900855275 (540h2m18.77193832s) [upload=473602]: Next returned nil segment Dec 28 03:31:18.900875082 (540h2m18.771958127s) [upload=473602]: Waiting for error group managing segments... Dec 28 03:31:19.183639524 (540h2m19.054722569s) [noise=true , address=145.79.227.17:4971 , piece_id=FBNJBASX5WPSGN2ZWD5OO455FUR2GB4HEOINSXVH5KDQKX5J3Y4A, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.227295391 (540h2m19.098378426s) [noise=true , address=145.79.227.5:4222 , piece_id=AEB6HPEMVMNAVVJ4OBPGD37WVFEYFMVEZTWWN7YFKZ3HZSUT6D4Q, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.230058483 (540h2m19.101141528s) [noise=true , address=145.79.227.11:4729 , piece_id=VWLRLCD75L4CY7GWINKD4GNB2IBPAKRHIJ7ZJLOXOSBOT773UMCQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.238000857 (540h2m19.109083892s) [noise=true , address=145.79.227.46:4603 , piece_id=BJ5UIOOWRPSV625DSQIQOG5SAJVAHCJUMSSN2WMTPG7ROTK2AUVA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.252059106 (540h2m19.123142151s) [noise=true , address=145.79.227.38:4977 , piece_id=4R6P6GRVIFFDNKYAO6DYVRXL6LPQBZY42EAYY4SIIJTTYLNTKPPQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.419378820 (540h2m19.290461855s) [noise=true , address=77.90.31.21:4877 , piece_id=ETKK3JSU72QSYPAYNTNHQRXHFFYEJFA46V5YXZ5PFPFYOK6WQNGQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.431951410 (540h2m19.303034445s) [noise=true , address=45.91.116.6:4974 , piece_id=W22IZ2D2BATH6E7YRVGU6C2ITD2N657F5JHGHSLV74TBYOVWGTYA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:19.466095857 (540h2m19.337178902s) [noise=true , address=45.91.116.5:4299 , piece_id=KOPWB3QKQL2YXKW63RENR5S6HUPHSTBIN2CVETQ2CMONY2AN5PFQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:20.128896220 (540h2m19.999979265s) [noise=true , address=45.91.116.14:4192 , piece_id=C5G7G6CQHJWIKJFE2GH76PKOKC73GM2A3ZXD5JEPNZB7SGRW7AQQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:20.348551019 (540h2m20.219634074s) [noise=true , address=145.79.227.42:4122 , piece_id=UKJ5MZBVCY7N7EUVCNJRTTZ5HROSMKNFKPGKRONPKSL5OMLGJROQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:20.370045229 (540h2m20.241128264s) [noise=true , address=145.79.227.41:4087 , piece_id=KD4LRWNGE67IN4FFMGEA2Z6MJSTDESZW5WSD5JLR6FRJWD4PKLXA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:20.372443494 (540h2m20.243526539s) [noise=true , address=145.79.227.4:4187 , piece_id=5RWXHVIVRJ3ZED7WQLYYJNJOFX2SBEKBFH5WAGNWQFPO4EBXOJWA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:20.420582297 (540h2m20.291665332s) [noise=true , address=145.79.227.36:4517 , piece_id=XNJGZJLNLXTMCA5BQ4MKD7M5XX2HTXRZ7LRMIAIXXZ5OTZOTEVMQ, seg_pos={0 0}, upload=473602]: Done uploading piece. err: <nil> Dec 28 03:31:20.420605401 (540h2m20.291688446s) [seg_pos={0 0}, upload=473602]: Segment reached optimal threshold of 13 pieces. Dec 28 03:31:20.420819248 (540h2m20.291902283s) [noise=true , address=145.79.227.33:4870 , piece_id=TLBT76FXM5I3I3VVJZDHNVMC2IU3Q57SBRXU6JHGBMPMCIEHTPOA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 03:31:20.420829317 (540h2m20.291912352s) [noise=true , address=145.79.227.33:4870 , piece_id=TLBT76FXM5I3I3VVJZDHNVMC2IU3Q57SBRXU6JHGBMPMCIEHTPOA, seg_pos={0 0}, upload=473602]: Long tail cancellation due to optimal threshold Dec 28 03:31:20.420871087 (540h2m20.291954152s) [noise=true , address=77.90.31.12:4590 , piece_id=BDO6LJTEQNFNDRLZ3FBPH6CALSYC44JINIGZZPCYGKOEGEGJ5YUA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 03:31:20.421097358 (540h2m20.292180433s) [noise=true , address=77.90.31.12:4590 , piece_id=BDO6LJTEQNFNDRLZ3FBPH6CALSYC44JINIGZZPCYGKOEGEGJ5YUA, seg_pos={0 0}, upload=473602]: Long tail cancellation due to optimal threshold Dec 28 03:31:20.421136492 (540h2m20.292219527s) [noise=true , address=77.90.31.10:4519 , piece_id=QCS44OTWILK6YLKZVW2XTUXXQPT3SP3CI4Z2DNVESJL4NNDFDYOA, seg_pos={0 0}, upload=473602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 28 03:31:20.421145229 (540h2m20.292228264s) [noise=true , address=77.90.31.10:4519 , piece_id=QCS44OTWILK6YLKZVW2XTUXXQPT3SP3CI4Z2DNVESJL4NNDFDYOA, seg_pos={0 0}, upload=473602]: Long tail cancellation due to optimal threshold Dec 28 03:31:20.421163594 (540h2m20.292246639s) [seg_pos={0 0}, upload=473602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:31:20.421234790 (540h2m20.292317835s) [upload=473602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:31:20.421240380 (540h2m20.292323425s) [upload=473602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:31:20.725355077 (540h2m20.596438122s) [upload=473602]: Done uploading segments... Dec 28 03:31:20.725380686 (540h2m20.596463721s) [upload=473602]: Upload finished. err: <nil> Duration: 30.003459975s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...X........X....X...........X.....XX.....X..X.X....X.XXXX..] duration: 5.001056927s trace: https://jaeger.dev.storj.io/trace/58e23bd7ac68184a FAIL (2025-12-28 03:31:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001082245s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.X] duration: 30.003648775s trace: https://jaeger.dev.storj.io/trace/7e4078f72ea52ee2 FAIL (2025-12-28 03:30:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:30:14.127988814 (540h1m13.999071849s) [upload=473586]: Starting upload Dec 28 03:30:14.128060130 (540h1m13.999143165s) [upload=473586]: Done starting upload Dec 28 03:30:14.128126827 (540h1m13.999209862s) [upload=473586]: Uploading segments... Dec 28 03:30:14.128386422 (540h1m13.999469457s) [upload=473586]: Got next segment. Inline: false Dec 28 03:30:14.128435725 (540h1m13.99951876s) [upload=473586]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:30:14.128441746 (540h1m13.999524801s) [upload=473586]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:30:14.371002152 (540h1m14.242085187s) [seg_pos={0 0}, upload=473586]: Begin upload segment... Dec 28 03:30:14.371047218 (540h1m14.242130253s) [seg_pos={0 0}, upload=473586]: Done begin upload segment. Dec 28 03:30:14.371074289 (540h1m14.242157325s) [noise=true , address=45.91.116.5:4297 , piece_id=AQPBT3IA7RLNRANLQB4MZE4YE5JJMCRT4KVZIVMKWDHRKGZS6ARA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.371168299 (540h1m14.242251344s) [noise=true , address=145.79.227.40:4049 , piece_id=W37QKPVYY4SKSKSFT3HGVTSHIR5LNDWR5FUNDDF4FQZCE7YU4OOQ, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.371797547 (540h1m14.242880592s) [noise=true , address=145.79.227.11:4730 , piece_id=6TGGW2LAB22JP6VVUVQLFBX73CLYIO7BHOKPPBXPPXQDE6CDEHFA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.372262432 (540h1m14.243345477s) [noise=true , address=145.79.227.42:4121 , piece_id=CDA4DKWJZIOYFZM3C3UD5IUTKFU6DKT7U2LJ3ZEJ5APRM7I5NZCQ, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.372868836 (540h1m14.243951881s) [noise=true , address=45.91.116.14:4192 , piece_id=NLQFTUDANYGYTBWJYTGVIXJRFKAF6N5HVFPFRHNMTK2DKSJEMN5A, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.373392323 (540h1m14.244475368s) [noise=true , address=45.91.116.10:4082 , piece_id=X6RXKOQD3PGLFLBJB3RP7NYGXBTBN73IDU7ZZNQQHY3B4HLHO3BA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.374696306 (540h1m14.245779341s) [noise=true , address=145.79.227.48:4303 , piece_id=UEN54CK2M36SCMJCTY6J5PASDY4VOYEOVVCNPLLLONHSGNNDJSDQ, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.375017949 (540h1m14.246100994s) [noise=true , address=77.90.31.21:4877 , piece_id=C22UA6OXDWJYYEXCDAFNQE3XQFKHWFTSBZ7ZHHEBU6AHSP7GZN4A, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.376011681 (540h1m14.247094726s) [noise=true , address=145.79.227.14:4839 , piece_id=VYYWAYVR6GGZSNCZHT67BF6AL2GIUD23AZURHA4P2TQ2DSW6H5MA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.376608798 (540h1m14.247691833s) [noise=true , address=145.79.227.52:4445 , piece_id=6P5UYUE6OYWPOY7T7FK7VMPK4L6JIF6T3SB55CSUD6O576TIOBGA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.377296027 (540h1m14.248379062s) [noise=true , address=77.90.31.19:4841 , piece_id=OCPN563QHFRZMPL3OHIBNYHX4XUDCNFHVOQ5H4OHN43DBNCACN6Q, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.378402864 (540h1m14.249485899s) [noise=true , address=145.79.227.39:4014 , piece_id=DH6RH3L3Z5BKXDXE4B5UXNE3WKAGXYYWFA6SYE5YBXZVH3E2JPCQ, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.378645656 (540h1m14.249728691s) [noise=true , address=145.79.227.13:4802 , piece_id=HO2B27B4UIHKNWLBWX7JCZO36DCKQR4CP3XR3NJQEQ2QV2I6D5BA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.380100086 (540h1m14.251183121s) [noise=true , address=145.79.227.45:4230 , piece_id=KOU5QR3PCQZX2JLCMS6NGC2IWQ675NNN5UR5N6BBKFFZD6MCPKBQ, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.381398618 (540h1m14.252481653s) [noise=true , address=145.79.227.46:4603 , piece_id=W6N5GOLZV2BRF2ISZUED4EACXEVOV55YF2RTMWR4ZDXMQMB6W6JQ, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:14.382652125 (540h1m14.25373517s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3LREQRVKW723SIKG3742PKUIEPI4BMYN65SJ7MBP455P3FGZRKA, seg_pos={0 0}, upload=473586]: Uploading piece... Dec 28 03:30:16.107159306 (540h1m15.978242341s) [upload=473586]: Next returned nil segment Dec 28 03:30:16.107171609 (540h1m15.978254654s) [upload=473586]: Waiting for error group managing segments... Dec 28 03:30:16.385571429 (540h1m16.256654464s) [noise=true , address=145.79.227.52:4445 , piece_id=6P5UYUE6OYWPOY7T7FK7VMPK4L6JIF6T3SB55CSUD6O576TIOBGA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.395050910 (540h1m16.266133945s) [noise=true , address=145.79.227.42:4121 , piece_id=CDA4DKWJZIOYFZM3C3UD5IUTKFU6DKT7U2LJ3ZEJ5APRM7I5NZCQ, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.397236322 (540h1m16.268319357s) [noise=true , address=145.79.227.13:4802 , piece_id=HO2B27B4UIHKNWLBWX7JCZO36DCKQR4CP3XR3NJQEQ2QV2I6D5BA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.402355039 (540h1m16.273438084s) [noise=true , address=145.79.227.11:4730 , piece_id=6TGGW2LAB22JP6VVUVQLFBX73CLYIO7BHOKPPBXPPXQDE6CDEHFA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.406992439 (540h1m16.278075484s) [noise=true , address=145.79.227.48:4303 , piece_id=UEN54CK2M36SCMJCTY6J5PASDY4VOYEOVVCNPLLLONHSGNNDJSDQ, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.450860311 (540h1m16.321943346s) [noise=true , address=145.79.227.45:4230 , piece_id=KOU5QR3PCQZX2JLCMS6NGC2IWQ675NNN5UR5N6BBKFFZD6MCPKBQ, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.613581969 (540h1m16.484665004s) [noise=true , address=45.91.116.14:4192 , piece_id=NLQFTUDANYGYTBWJYTGVIXJRFKAF6N5HVFPFRHNMTK2DKSJEMN5A, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.615171807 (540h1m16.486254842s) [noise=true , address=45.91.116.5:4297 , piece_id=AQPBT3IA7RLNRANLQB4MZE4YE5JJMCRT4KVZIVMKWDHRKGZS6ARA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.695444719 (540h1m16.566527765s) [noise=true , address=77.90.31.19:4841 , piece_id=OCPN563QHFRZMPL3OHIBNYHX4XUDCNFHVOQ5H4OHN43DBNCACN6Q, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:16.746711380 (540h1m16.617794425s) [noise=true , address=45.91.116.10:4082 , piece_id=X6RXKOQD3PGLFLBJB3RP7NYGXBTBN73IDU7ZZNQQHY3B4HLHO3BA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:17.340006754 (540h1m17.211089789s) [noise=true , address=145.79.227.46:4603 , piece_id=W6N5GOLZV2BRF2ISZUED4EACXEVOV55YF2RTMWR4ZDXMQMB6W6JQ, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:17.459891214 (540h1m17.33097426s) [noise=true , address=145.79.227.14:4839 , piece_id=VYYWAYVR6GGZSNCZHT67BF6AL2GIUD23AZURHA4P2TQ2DSW6H5MA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:17.483845827 (540h1m17.354928862s) [noise=true , address=145.79.227.39:4014 , piece_id=DH6RH3L3Z5BKXDXE4B5UXNE3WKAGXYYWFA6SYE5YBXZVH3E2JPCQ, seg_pos={0 0}, upload=473586]: Done uploading piece. err: <nil> Dec 28 03:30:17.483867088 (540h1m17.354950123s) [seg_pos={0 0}, upload=473586]: Segment reached optimal threshold of 13 pieces. Dec 28 03:30:17.484150447 (540h1m17.355233492s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3LREQRVKW723SIKG3742PKUIEPI4BMYN65SJ7MBP455P3FGZRKA, seg_pos={0 0}, upload=473586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: expected piece hash; context canceled; EOF Dec 28 03:30:17.484165517 (540h1m17.355248562s) [noise=true , address=145.79.227.5:4223 , piece_id=Y3LREQRVKW723SIKG3742PKUIEPI4BMYN65SJ7MBP455P3FGZRKA, seg_pos={0 0}, upload=473586]: Long tail cancellation due to optimal threshold Dec 28 03:30:17.484387709 (540h1m17.355470754s) [noise=true , address=77.90.31.21:4877 , piece_id=C22UA6OXDWJYYEXCDAFNQE3XQFKHWFTSBZ7ZHHEBU6AHSP7GZN4A, seg_pos={0 0}, upload=473586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:56696->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:56696->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56696->77.90.31.21:4877: use of closed network connection Dec 28 03:30:17.484394672 (540h1m17.355477707s) [noise=true , address=77.90.31.21:4877 , piece_id=C22UA6OXDWJYYEXCDAFNQE3XQFKHWFTSBZ7ZHHEBU6AHSP7GZN4A, seg_pos={0 0}, upload=473586]: Long tail cancellation due to optimal threshold Dec 28 03:30:17.484591668 (540h1m17.355674713s) [noise=true , address=145.79.227.40:4049 , piece_id=W37QKPVYY4SKSKSFT3HGVTSHIR5LNDWR5FUNDDF4FQZCE7YU4OOQ, seg_pos={0 0}, upload=473586]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 28 03:30:17.484597759 (540h1m17.355680794s) [noise=true , address=145.79.227.40:4049 , piece_id=W37QKPVYY4SKSKSFT3HGVTSHIR5LNDWR5FUNDDF4FQZCE7YU4OOQ, seg_pos={0 0}, upload=473586]: Long tail cancellation due to optimal threshold Dec 28 03:30:17.484613048 (540h1m17.355696083s) [seg_pos={0 0}, upload=473586]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:30:17.484677330 (540h1m17.355760366s) [upload=473586]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:30:17.484682350 (540h1m17.355765385s) [upload=473586]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:30:17.775856886 (540h1m17.646939921s) [upload=473586]: Done uploading segments... Dec 28 03:30:17.775875811 (540h1m17.646958847s) [upload=473586]: Upload finished. err: <nil> Duration: 30.003671006s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 40.475338367s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:30:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 40.475364306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X........X....X...........X.....XX.....X..X.X....X.XXXX.] duration: 5.001155837s trace: https://jaeger.dev.storj.io/trace/58e23bd7ac68184a FAIL (2025-12-28 03:30:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001182547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....X........X....X...........X.....XX.....X..X.X....X.XXXX] duration: 5.001497268s trace: https://jaeger.dev.storj.io/trace/58e23bd7ac68184a FAIL (2025-12-28 03:29:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001531774s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 59.619874168s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:29:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 59.6198968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.] duration: 14.643838579s trace: https://jaeger.dev.storj.io/trace/543887e071cd0ad6 FAIL (2025-12-28 03:29:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: G7QN5S2PSUZIZZRWDJ57DHDWSQZFI6IANAYJFVXEKEIXO5FTHQGA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: MJEV6IHJ6LYWBTOU7EV6GFLI37H2FLD7Z2M77KBT63NSQE3AOTMA) context deadline exceeded ======LOGS====== Dec 28 03:28:09.940038461 (539h59m9.811121496s) [upload=473555]: Starting upload Dec 28 03:28:09.940120238 (539h59m9.811203273s) [upload=473555]: Done starting upload Dec 28 03:28:09.940278819 (539h59m9.811361864s) [upload=473555]: Uploading segments... Dec 28 03:28:09.940380352 (539h59m9.811463387s) [upload=473555]: Got next segment. Inline: false Dec 28 03:28:09.940442691 (539h59m9.811525726s) [upload=473555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:28:09.940450887 (539h59m9.811533922s) [upload=473555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:28:10.183839249 (539h59m10.054922294s) [seg_pos={0 0}, upload=473555]: Begin upload segment... Dec 28 03:28:10.183899944 (539h59m10.054982979s) [seg_pos={0 0}, upload=473555]: Done begin upload segment. Dec 28 03:28:10.183916977 (539h59m10.055000022s) [noise=true , address=45.91.116.4:4939 , piece_id=CAI4BI775ZCTD42S4D4I6CYCALWWAFZCC7QSUXEN62N47OTZS3UQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.183934811 (539h59m10.055017856s) [noise=true , address=145.79.227.3:4150 , piece_id=D2WKKYNUMXTDKM3YT7PMT4XM5KAIWFK25OOWWDLHEAKBOWZ6IF6A, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.184589658 (539h59m10.055672693s) [noise=true , address=77.90.31.21:4880 , piece_id=DCNVQPMKGURRRDGLHVOQQF5YA4HHKTFVQ47ZCRZFFQMTG6Y75OYA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.185110179 (539h59m10.056193224s) [noise=true , address=77.90.31.18:4808 , piece_id=O4IPTU2VOFPJATF3LSIGCD424K4FGOFPXR7WQ3ZGUC75TKEAIVNA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.185483180 (539h59m10.056566215s) [noise=true , address=45.91.116.13:4155 , piece_id=VA2DP5YCOFO3BS3ZP4BNJGK52JC6C2E6KONX5EVHICHLLSUQVGVA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.185922607 (539h59m10.057005652s) [noise=true , address=77.90.31.10:4517 , piece_id=A2EEF4VWBV5K2ZVFZCVOSP7CKKI5QH3HYHW2EH6UZFBKAQYX7XPQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.186351613 (539h59m10.057434658s) [noise=true , address=145.79.227.37:4566 , piece_id=VQ753J6UDJLUEYI7YRJZ6ZBUZT4ZBEBLKC6QI526X2OKUNZ4OKSA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.186986362 (539h59m10.058069407s) [noise=true , address=45.91.116.14:4190 , piece_id=ODODOBA22HJZF6CH2DPXECRFYD2NSCEQNAUP6LTEM2ZQ7Z3QBSUA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.187462839 (539h59m10.058545884s) [noise=true , address=77.90.31.11:4554 , piece_id=QEMUOMQ4CN2ZL66PTQ34OS6S2KMVGBBUOYJ2ILJKMUK2XMZHDONA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.187947051 (539h59m10.059030096s) [noise=true , address=145.79.227.6:4257 , piece_id=XGU5Z6UGUQOZQSF5JLCLPPFDFZS3FCTDGY3MFQGCZQVF7PTW4FAQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.189596121 (539h59m10.060679166s) [noise=true , address=45.91.116.10:4083 , piece_id=G7QN5S2PSUZIZZRWDJ57DHDWSQZFI6IANAYJFVXEKEIXO5FTHQGA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.191257725 (539h59m10.06234076s) [noise=true , address=45.91.116.9:4262 , piece_id=MJEV6IHJ6LYWBTOU7EV6GFLI37H2FLD7Z2M77KBT63NSQE3AOTMA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.192982429 (539h59m10.064065474s) [noise=true , address=145.79.227.24:4583 , piece_id=PBEZB2HV3MM777HAIZ3K2NBMIU6KHE2KBTONUIRLEAPGMHAKFDTA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.194763050 (539h59m10.065846095s) [noise=true , address=77.90.31.17:4770 , piece_id=BFUI3UXNOD3AEMJNVP7WII5CUU54VGYYCIEUNNWLA4XNJIW6FRWQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.196424423 (539h59m10.067507458s) [noise=true , address=145.79.227.12:4765 , piece_id=QHQ52FIBYL4EXOWOCDKO5DSQQWJ2XCBWZA2W347OACUVJHCYEJZQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.198067191 (539h59m10.069150236s) [noise=true , address=145.79.227.36:4519 , piece_id=L7P3WDEPZLWSHD2GJSH2IQ33ZEKPWEVU6E5TU32FBCTHEG2CYUSA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:11.909500356 (539h59m11.780583402s) [upload=473555]: Next returned nil segment Dec 28 03:28:11.909520135 (539h59m11.78060317s) [upload=473555]: Waiting for error group managing segments... Dec 28 03:28:12.208083810 (539h59m12.079166845s) [noise=true , address=145.79.227.12:4765 , piece_id=QHQ52FIBYL4EXOWOCDKO5DSQQWJ2XCBWZA2W347OACUVJHCYEJZQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.211684195 (539h59m12.08276723s) [noise=true , address=145.79.227.3:4150 , piece_id=D2WKKYNUMXTDKM3YT7PMT4XM5KAIWFK25OOWWDLHEAKBOWZ6IF6A, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.336462514 (539h59m12.207545549s) [noise=true , address=145.79.227.24:4583 , piece_id=PBEZB2HV3MM777HAIZ3K2NBMIU6KHE2KBTONUIRLEAPGMHAKFDTA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.361562228 (539h59m12.232645263s) [noise=true , address=77.90.31.17:4770 , piece_id=BFUI3UXNOD3AEMJNVP7WII5CUU54VGYYCIEUNNWLA4XNJIW6FRWQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.380992837 (539h59m12.252075872s) [noise=true , address=77.90.31.21:4880 , piece_id=DCNVQPMKGURRRDGLHVOQQF5YA4HHKTFVQ47ZCRZFFQMTG6Y75OYA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.458972574 (539h59m12.330055609s) [noise=true , address=45.91.116.10:4083 , piece_id=G7QN5S2PSUZIZZRWDJ57DHDWSQZFI6IANAYJFVXEKEIXO5FTHQGA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.490659090 (539h59m12.361742125s) [noise=true , address=77.90.31.10:4517 , piece_id=A2EEF4VWBV5K2ZVFZCVOSP7CKKI5QH3HYHW2EH6UZFBKAQYX7XPQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.503349425 (539h59m12.37443246s) [noise=true , address=77.90.31.11:4554 , piece_id=QEMUOMQ4CN2ZL66PTQ34OS6S2KMVGBBUOYJ2ILJKMUK2XMZHDONA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.504343437 (539h59m12.375426472s) [noise=true , address=45.91.116.14:4190 , piece_id=ODODOBA22HJZF6CH2DPXECRFYD2NSCEQNAUP6LTEM2ZQ7Z3QBSUA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.522261727 (539h59m12.393344762s) [noise=true , address=45.91.116.13:4155 , piece_id=VA2DP5YCOFO3BS3ZP4BNJGK52JC6C2E6KONX5EVHICHLLSUQVGVA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.295056487 (539h59m13.166139522s) [noise=true , address=45.91.116.9:4262 , piece_id=MJEV6IHJ6LYWBTOU7EV6GFLI37H2FLD7Z2M77KBT63NSQE3AOTMA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.317442915 (539h59m13.188525961s) [noise=true , address=145.79.227.6:4257 , piece_id=XGU5Z6UGUQOZQSF5JLCLPPFDFZS3FCTDGY3MFQGCZQVF7PTW4FAQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.460656899 (539h59m13.331739934s) [noise=true , address=45.91.116.4:4939 , piece_id=CAI4BI775ZCTD42S4D4I6CYCALWWAFZCC7QSUXEN62N47OTZS3UQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.460675874 (539h59m13.331758909s) [seg_pos={0 0}, upload=473555]: Segment reached optimal threshold of 13 pieces. Dec 28 03:28:13.460852200 (539h59m13.331935235s) [noise=true , address=145.79.227.36:4519 , piece_id=L7P3WDEPZLWSHD2GJSH2IQ33ZEKPWEVU6E5TU32FBCTHEG2CYUSA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 28 03:28:13.460860445 (539h59m13.33194348s) [noise=true , address=145.79.227.36:4519 , piece_id=L7P3WDEPZLWSHD2GJSH2IQ33ZEKPWEVU6E5TU32FBCTHEG2CYUSA, seg_pos={0 0}, upload=473555]: Long tail cancellation due to optimal threshold Dec 28 03:28:13.461047913 (539h59m13.332130958s) [noise=true , address=77.90.31.18:4808 , piece_id=O4IPTU2VOFPJATF3LSIGCD424K4FGOFPXR7WQ3ZGUC75TKEAIVNA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:47920->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:47920->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47920->77.90.31.18:4808: use of closed network connection Dec 28 03:28:13.461078942 (539h59m13.332161977s) [noise=true , address=77.90.31.18:4808 , piece_id=O4IPTU2VOFPJATF3LSIGCD424K4FGOFPXR7WQ3ZGUC75TKEAIVNA, seg_pos={0 0}, upload=473555]: Long tail cancellation due to optimal threshold Dec 28 03:28:13.461099100 (539h59m13.332182135s) [noise=true , address=145.79.227.37:4566 , piece_id=VQ753J6UDJLUEYI7YRJZ6ZBUZT4ZBEBLKC6QI526X2OKUNZ4OKSA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 28 03:28:13.461106965 (539h59m13.33219001s) [noise=true , address=145.79.227.37:4566 , piece_id=VQ753J6UDJLUEYI7YRJZ6ZBUZT4ZBEBLKC6QI526X2OKUNZ4OKSA, seg_pos={0 0}, upload=473555]: Long tail cancellation due to optimal threshold Dec 28 03:28:13.461131311 (539h59m13.332214346s) [seg_pos={0 0}, upload=473555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:28:13.461201926 (539h59m13.332284961s) [upload=473555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:28:13.461206865 (539h59m13.3322899s) [upload=473555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:28:13.757488706 (539h59m13.628571741s) [upload=473555]: Done uploading segments... Dec 28 03:28:13.757506760 (539h59m13.628589805s) [upload=473555]: Upload finished. err: <nil> Duration: 14.64386547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X........X....X...........X.....XX.....X..X.X....X.XXX] duration: 5.001904253s trace: https://jaeger.dev.storj.io/trace/4544ac0b3afb716d FAIL (2025-12-28 03:29:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001927197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 32.962380651s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:28:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 32.962413003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX] duration: 30.001988081s trace: https://jaeger.dev.storj.io/trace/543887e071cd0ad6 FAIL (2025-12-28 03:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: G7QN5S2PSUZIZZRWDJ57DHDWSQZFI6IANAYJFVXEKEIXO5FTHQGA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: MJEV6IHJ6LYWBTOU7EV6GFLI37H2FLD7Z2M77KBT63NSQE3AOTMA) context deadline exceeded ======LOGS====== Dec 28 03:28:09.940038461 (539h59m9.811121496s) [upload=473555]: Starting upload Dec 28 03:28:09.940120238 (539h59m9.811203273s) [upload=473555]: Done starting upload Dec 28 03:28:09.940278819 (539h59m9.811361864s) [upload=473555]: Uploading segments... Dec 28 03:28:09.940380352 (539h59m9.811463387s) [upload=473555]: Got next segment. Inline: false Dec 28 03:28:09.940442691 (539h59m9.811525726s) [upload=473555]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:28:09.940450887 (539h59m9.811533922s) [upload=473555]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:28:10.183839249 (539h59m10.054922294s) [seg_pos={0 0}, upload=473555]: Begin upload segment... Dec 28 03:28:10.183899944 (539h59m10.054982979s) [seg_pos={0 0}, upload=473555]: Done begin upload segment. Dec 28 03:28:10.183916977 (539h59m10.055000022s) [noise=true , address=45.91.116.4:4939 , piece_id=CAI4BI775ZCTD42S4D4I6CYCALWWAFZCC7QSUXEN62N47OTZS3UQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.183934811 (539h59m10.055017856s) [noise=true , address=145.79.227.3:4150 , piece_id=D2WKKYNUMXTDKM3YT7PMT4XM5KAIWFK25OOWWDLHEAKBOWZ6IF6A, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.184589658 (539h59m10.055672693s) [noise=true , address=77.90.31.21:4880 , piece_id=DCNVQPMKGURRRDGLHVOQQF5YA4HHKTFVQ47ZCRZFFQMTG6Y75OYA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.185110179 (539h59m10.056193224s) [noise=true , address=77.90.31.18:4808 , piece_id=O4IPTU2VOFPJATF3LSIGCD424K4FGOFPXR7WQ3ZGUC75TKEAIVNA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.185483180 (539h59m10.056566215s) [noise=true , address=45.91.116.13:4155 , piece_id=VA2DP5YCOFO3BS3ZP4BNJGK52JC6C2E6KONX5EVHICHLLSUQVGVA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.185922607 (539h59m10.057005652s) [noise=true , address=77.90.31.10:4517 , piece_id=A2EEF4VWBV5K2ZVFZCVOSP7CKKI5QH3HYHW2EH6UZFBKAQYX7XPQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.186351613 (539h59m10.057434658s) [noise=true , address=145.79.227.37:4566 , piece_id=VQ753J6UDJLUEYI7YRJZ6ZBUZT4ZBEBLKC6QI526X2OKUNZ4OKSA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.186986362 (539h59m10.058069407s) [noise=true , address=45.91.116.14:4190 , piece_id=ODODOBA22HJZF6CH2DPXECRFYD2NSCEQNAUP6LTEM2ZQ7Z3QBSUA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.187462839 (539h59m10.058545884s) [noise=true , address=77.90.31.11:4554 , piece_id=QEMUOMQ4CN2ZL66PTQ34OS6S2KMVGBBUOYJ2ILJKMUK2XMZHDONA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.187947051 (539h59m10.059030096s) [noise=true , address=145.79.227.6:4257 , piece_id=XGU5Z6UGUQOZQSF5JLCLPPFDFZS3FCTDGY3MFQGCZQVF7PTW4FAQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.189596121 (539h59m10.060679166s) [noise=true , address=45.91.116.10:4083 , piece_id=G7QN5S2PSUZIZZRWDJ57DHDWSQZFI6IANAYJFVXEKEIXO5FTHQGA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.191257725 (539h59m10.06234076s) [noise=true , address=45.91.116.9:4262 , piece_id=MJEV6IHJ6LYWBTOU7EV6GFLI37H2FLD7Z2M77KBT63NSQE3AOTMA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.192982429 (539h59m10.064065474s) [noise=true , address=145.79.227.24:4583 , piece_id=PBEZB2HV3MM777HAIZ3K2NBMIU6KHE2KBTONUIRLEAPGMHAKFDTA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.194763050 (539h59m10.065846095s) [noise=true , address=77.90.31.17:4770 , piece_id=BFUI3UXNOD3AEMJNVP7WII5CUU54VGYYCIEUNNWLA4XNJIW6FRWQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.196424423 (539h59m10.067507458s) [noise=true , address=145.79.227.12:4765 , piece_id=QHQ52FIBYL4EXOWOCDKO5DSQQWJ2XCBWZA2W347OACUVJHCYEJZQ, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:10.198067191 (539h59m10.069150236s) [noise=true , address=145.79.227.36:4519 , piece_id=L7P3WDEPZLWSHD2GJSH2IQ33ZEKPWEVU6E5TU32FBCTHEG2CYUSA, seg_pos={0 0}, upload=473555]: Uploading piece... Dec 28 03:28:11.909500356 (539h59m11.780583402s) [upload=473555]: Next returned nil segment Dec 28 03:28:11.909520135 (539h59m11.78060317s) [upload=473555]: Waiting for error group managing segments... Dec 28 03:28:12.208083810 (539h59m12.079166845s) [noise=true , address=145.79.227.12:4765 , piece_id=QHQ52FIBYL4EXOWOCDKO5DSQQWJ2XCBWZA2W347OACUVJHCYEJZQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.211684195 (539h59m12.08276723s) [noise=true , address=145.79.227.3:4150 , piece_id=D2WKKYNUMXTDKM3YT7PMT4XM5KAIWFK25OOWWDLHEAKBOWZ6IF6A, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.336462514 (539h59m12.207545549s) [noise=true , address=145.79.227.24:4583 , piece_id=PBEZB2HV3MM777HAIZ3K2NBMIU6KHE2KBTONUIRLEAPGMHAKFDTA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.361562228 (539h59m12.232645263s) [noise=true , address=77.90.31.17:4770 , piece_id=BFUI3UXNOD3AEMJNVP7WII5CUU54VGYYCIEUNNWLA4XNJIW6FRWQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.380992837 (539h59m12.252075872s) [noise=true , address=77.90.31.21:4880 , piece_id=DCNVQPMKGURRRDGLHVOQQF5YA4HHKTFVQ47ZCRZFFQMTG6Y75OYA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.458972574 (539h59m12.330055609s) [noise=true , address=45.91.116.10:4083 , piece_id=G7QN5S2PSUZIZZRWDJ57DHDWSQZFI6IANAYJFVXEKEIXO5FTHQGA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.490659090 (539h59m12.361742125s) [noise=true , address=77.90.31.10:4517 , piece_id=A2EEF4VWBV5K2ZVFZCVOSP7CKKI5QH3HYHW2EH6UZFBKAQYX7XPQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.503349425 (539h59m12.37443246s) [noise=true , address=77.90.31.11:4554 , piece_id=QEMUOMQ4CN2ZL66PTQ34OS6S2KMVGBBUOYJ2ILJKMUK2XMZHDONA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.504343437 (539h59m12.375426472s) [noise=true , address=45.91.116.14:4190 , piece_id=ODODOBA22HJZF6CH2DPXECRFYD2NSCEQNAUP6LTEM2ZQ7Z3QBSUA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:12.522261727 (539h59m12.393344762s) [noise=true , address=45.91.116.13:4155 , piece_id=VA2DP5YCOFO3BS3ZP4BNJGK52JC6C2E6KONX5EVHICHLLSUQVGVA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.295056487 (539h59m13.166139522s) [noise=true , address=45.91.116.9:4262 , piece_id=MJEV6IHJ6LYWBTOU7EV6GFLI37H2FLD7Z2M77KBT63NSQE3AOTMA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.317442915 (539h59m13.188525961s) [noise=true , address=145.79.227.6:4257 , piece_id=XGU5Z6UGUQOZQSF5JLCLPPFDFZS3FCTDGY3MFQGCZQVF7PTW4FAQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.460656899 (539h59m13.331739934s) [noise=true , address=45.91.116.4:4939 , piece_id=CAI4BI775ZCTD42S4D4I6CYCALWWAFZCC7QSUXEN62N47OTZS3UQ, seg_pos={0 0}, upload=473555]: Done uploading piece. err: <nil> Dec 28 03:28:13.460675874 (539h59m13.331758909s) [seg_pos={0 0}, upload=473555]: Segment reached optimal threshold of 13 pieces. Dec 28 03:28:13.460852200 (539h59m13.331935235s) [noise=true , address=145.79.227.36:4519 , piece_id=L7P3WDEPZLWSHD2GJSH2IQ33ZEKPWEVU6E5TU32FBCTHEG2CYUSA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: expected piece hash; context canceled; EOF Dec 28 03:28:13.460860445 (539h59m13.33194348s) [noise=true , address=145.79.227.36:4519 , piece_id=L7P3WDEPZLWSHD2GJSH2IQ33ZEKPWEVU6E5TU32FBCTHEG2CYUSA, seg_pos={0 0}, upload=473555]: Long tail cancellation due to optimal threshold Dec 28 03:28:13.461047913 (539h59m13.332130958s) [noise=true , address=77.90.31.18:4808 , piece_id=O4IPTU2VOFPJATF3LSIGCD424K4FGOFPXR7WQ3ZGUC75TKEAIVNA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:47920->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:47920->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47920->77.90.31.18:4808: use of closed network connection Dec 28 03:28:13.461078942 (539h59m13.332161977s) [noise=true , address=77.90.31.18:4808 , piece_id=O4IPTU2VOFPJATF3LSIGCD424K4FGOFPXR7WQ3ZGUC75TKEAIVNA, seg_pos={0 0}, upload=473555]: Long tail cancellation due to optimal threshold Dec 28 03:28:13.461099100 (539h59m13.332182135s) [noise=true , address=145.79.227.37:4566 , piece_id=VQ753J6UDJLUEYI7YRJZ6ZBUZT4ZBEBLKC6QI526X2OKUNZ4OKSA, seg_pos={0 0}, upload=473555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 28 03:28:13.461106965 (539h59m13.33219001s) [noise=true , address=145.79.227.37:4566 , piece_id=VQ753J6UDJLUEYI7YRJZ6ZBUZT4ZBEBLKC6QI526X2OKUNZ4OKSA, seg_pos={0 0}, upload=473555]: Long tail cancellation due to optimal threshold Dec 28 03:28:13.461131311 (539h59m13.332214346s) [seg_pos={0 0}, upload=473555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:28:13.461201926 (539h59m13.332284961s) [upload=473555]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:28:13.461206865 (539h59m13.3322899s) [upload=473555]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:28:13.757488706 (539h59m13.628571741s) [upload=473555]: Done uploading segments... Dec 28 03:28:13.757506760 (539h59m13.628589805s) [upload=473555]: Upload finished. err: <nil> Duration: 30.001999342s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXX] duration: 30.003699795s trace: https://jaeger.dev.storj.io/trace/147e1a4c2669889a FAIL (2025-12-28 03:28:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ec 28 03:27:39.710643636 (539h58m39.581726681s) [upload=473546]: Starting upload Dec 28 03:27:39.710725803 (539h58m39.581808848s) [upload=473546]: Done starting upload Dec 28 03:27:39.710856542 (539h58m39.581939577s) [upload=473546]: Uploading segments... Dec 28 03:27:39.711148298 (539h58m39.582231343s) [upload=473546]: Got next segment. Inline: false Dec 28 03:27:39.711193353 (539h58m39.582276388s) [upload=473546]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:27:39.711200196 (539h58m39.582283251s) [upload=473546]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:27:39.963234021 (539h58m39.834317056s) [seg_pos={0 0}, upload=473546]: Begin upload segment... Dec 28 03:27:39.963275179 (539h58m39.834358214s) [seg_pos={0 0}, upload=473546]: Done begin upload segment. Dec 28 03:27:39.963290879 (539h58m39.834373924s) [noise=true , address=45.91.116.10:4082 , piece_id=E2B7KZQDXKZZGCWXSQJYI63SFYI3LBRLLFJWHJ4LGJLZJ65DVBLQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.964191263 (539h58m39.835274298s) [noise=true , address=145.79.227.13:4804 , piece_id=7NZPP6VLOQ7FSLIESWJVDUCH345N2E742LDQOJUBFGLWPHNUSQFA, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.964572279 (539h58m39.835655314s) [noise=true , address=45.91.116.7:4011 , piece_id=45TSWIKS3RYS7KPYS5OIMMIRFJHUKOZF2HAGPTHKB7ELHUOQKVSA, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.964896537 (539h58m39.835979572s) [noise=true , address=145.79.227.14:4837 , piece_id=CWJJFSEDAOVKSHRZ6DSUWI4BQR5ZWJMPAPW6ZH2H72NZM5F6XXVQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.965202298 (539h58m39.836285333s) [noise=true , address=145.79.227.7:4293 , piece_id=ZEAPXOZZU7IWBQJSWOMB7XLT2NNLA656AQ6VE3ZIP2VFL73LY5DA, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.965510165 (539h58m39.8365932s) [noise=true , address=45.91.116.8:4045 , piece_id=64IIJ7L4G7I6AAGXYA45O2GPZFAIV4PFLXBFS7P6ULA5BTK6I5NA, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.965902652 (539h58m39.836985687s) [noise=true , address=145.79.227.32:4836 , piece_id=755QKLZM2F47S7VD6GZWZFF4OOXYER5TABEX3PKYPOP7FZC6UWHQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.966224455 (539h58m39.83730749s) [noise=true , address=45.91.116.4:4939 , piece_id=QDMEKHM2SN4US2IEQNWOLUXJLDKTFKGRPJMVEWZVJM3CQXAWDMFQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.967286317 (539h58m39.838369382s) [noise=true , address=45.91.116.12:4225 , piece_id=WNRGM6FIR263KUDQGQJZBQMWYEARC6QCYYUYJCGZ4MZ5SHD62GEQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.968350413 (539h58m39.839433448s) [noise=true , address=145.79.227.2:4115 , piece_id=OAHVUJQJFAMQCV7HA4BV3OWVJASMF5A2XLZUDUBCPLNGASITXXYA, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.969035076 (539h58m39.840118122s) [noise=true , address=145.79.227.6:4259 , piece_id=5JMPCKF4NC5S7EOEQDCVVBZOO3D4JWK5ZYI62IX6MPFSUNSCJTWQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.969681628 (539h58m39.840764663s) [noise=true , address=145.79.227.47:4267 , piece_id=C47EDOMIV77YYDCJTNZ2JMXVP3RT7AL6F67BYNWDZWOVN6DNIWEA, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.970304394 (539h58m39.841387429s) [noise=true , address=145.79.227.43:4159 , piece_id=F3VBZPCAJGYMG2P6XDB3NT5EUUC2IP52OPMCAHGUU6EHDK72436A, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.970813985 (539h58m39.84189702s) [noise=true , address=145.79.227.22:4511 , piece_id=FBMTQ7W6C555ULKTYZ6CJGN7OISS3ORJHFYNBNWQH5JF5JERQE7A, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.971264002 (539h58m39.842347037s) [noise=true , address=145.79.227.15:4874 , piece_id=3U2M6UO77BTVNWNHXUOV27Z4KES7MK2FDVGHJ24EO7X6US4VQYKQ, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:39.971745207 (539h58m39.842828242s) [noise=true , address=77.90.31.10:4518 , piece_id=WGLDQCEMEJSOOLMM2CGSA54MZD7GQ3VHJFC5TDJJN5XSWTLWTG3Q, seg_pos={0 0}, upload=473546]: Uploading piece... Dec 28 03:27:41.777359629 (539h58m41.648442674s) [upload=473546]: Next returned nil segment Dec 28 03:27:41.777376832 (539h58m41.648459877s) [upload=473546]: Waiting for error group managing segments... Dec 28 03:27:42.051060231 (539h58m41.922143266s) [noise=true , address=145.79.227.22:4511 , piece_id=FBMTQ7W6C555ULKTYZ6CJGN7OISS3ORJHFYNBNWQH5JF5JERQE7A, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.073608125 (539h58m41.94469116s) [noise=true , address=145.79.227.13:4804 , piece_id=7NZPP6VLOQ7FSLIESWJVDUCH345N2E742LDQOJUBFGLWPHNUSQFA, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.158028656 (539h58m42.029111692s) [noise=true , address=145.79.227.7:4293 , piece_id=ZEAPXOZZU7IWBQJSWOMB7XLT2NNLA656AQ6VE3ZIP2VFL73LY5DA, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.172109930 (539h58m42.043192965s) [noise=true , address=145.79.227.32:4836 , piece_id=755QKLZM2F47S7VD6GZWZFF4OOXYER5TABEX3PKYPOP7FZC6UWHQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.177076567 (539h58m42.048159602s) [noise=true , address=77.90.31.10:4518 , piece_id=WGLDQCEMEJSOOLMM2CGSA54MZD7GQ3VHJFC5TDJJN5XSWTLWTG3Q, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.223673265 (539h58m42.0947563s) [noise=true , address=45.91.116.8:4045 , piece_id=64IIJ7L4G7I6AAGXYA45O2GPZFAIV4PFLXBFS7P6ULA5BTK6I5NA, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.235182671 (539h58m42.106265706s) [noise=true , address=45.91.116.12:4225 , piece_id=WNRGM6FIR263KUDQGQJZBQMWYEARC6QCYYUYJCGZ4MZ5SHD62GEQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.241772269 (539h58m42.112855314s) [noise=true , address=45.91.116.4:4939 , piece_id=QDMEKHM2SN4US2IEQNWOLUXJLDKTFKGRPJMVEWZVJM3CQXAWDMFQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.263437763 (539h58m42.134520808s) [noise=true , address=45.91.116.7:4011 , piece_id=45TSWIKS3RYS7KPYS5OIMMIRFJHUKOZF2HAGPTHKB7ELHUOQKVSA, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.289962490 (539h58m42.161045545s) [noise=true , address=145.79.227.14:4837 , piece_id=CWJJFSEDAOVKSHRZ6DSUWI4BQR5ZWJMPAPW6ZH2H72NZM5F6XXVQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.604560648 (539h58m42.475643683s) [noise=true , address=45.91.116.10:4082 , piece_id=E2B7KZQDXKZZGCWXSQJYI63SFYI3LBRLLFJWHJ4LGJLZJ65DVBLQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.949332000 (539h58m42.820415035s) [noise=true , address=145.79.227.15:4874 , piece_id=3U2M6UO77BTVNWNHXUOV27Z4KES7MK2FDVGHJ24EO7X6US4VQYKQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.966522845 (539h58m42.83760588s) [noise=true , address=145.79.227.2:4115 , piece_id=OAHVUJQJFAMQCV7HA4BV3OWVJASMF5A2XLZUDUBCPLNGASITXXYA, seg_pos={0 0}, upload=473546]: Done uploading piece. err: <nil> Dec 28 03:27:42.966572991 (539h58m42.837656036s) [seg_pos={0 0}, upload=473546]: Segment reached optimal threshold of 13 pieces. Dec 28 03:27:42.966898360 (539h58m42.837981405s) [noise=true , address=145.79.227.6:4259 , piece_id=5JMPCKF4NC5S7EOEQDCVVBZOO3D4JWK5ZYI62IX6MPFSUNSCJTWQ, seg_pos={0 0}, upload=473546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y): protocol: write tcp 188.166.225.170:44658->145.79.227.6:4259: use of closed network connection; write tcp 188.166.225.170:44658->145.79.227.6:4259: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44658->145.79.227.6:4259: use of closed network connection Dec 28 03:27:42.966907457 (539h58m42.837990502s) [noise=true , address=145.79.227.6:4259 , piece_id=5JMPCKF4NC5S7EOEQDCVVBZOO3D4JWK5ZYI62IX6MPFSUNSCJTWQ, seg_pos={0 0}, upload=473546]: Long tail cancellation due to optimal threshold Dec 28 03:27:42.966945690 (539h58m42.838028725s) [noise=true , address=145.79.227.43:4159 , piece_id=F3VBZPCAJGYMG2P6XDB3NT5EUUC2IP52OPMCAHGUU6EHDK72436A, seg_pos={0 0}, upload=473546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 03:27:42.966961941 (539h58m42.838044976s) [noise=true , address=145.79.227.43:4159 , piece_id=F3VBZPCAJGYMG2P6XDB3NT5EUUC2IP52OPMCAHGUU6EHDK72436A, seg_pos={0 0}, upload=473546]: Long tail cancellation due to optimal threshold Dec 28 03:27:42.967087259 (539h58m42.838170294s) [noise=true , address=145.79.227.47:4267 , piece_id=C47EDOMIV77YYDCJTNZ2JMXVP3RT7AL6F67BYNWDZWOVN6DNIWEA, seg_pos={0 0}, upload=473546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF Dec 28 03:27:42.967094153 (539h58m42.838177188s) [noise=true , address=145.79.227.47:4267 , piece_id=C47EDOMIV77YYDCJTNZ2JMXVP3RT7AL6F67BYNWDZWOVN6DNIWEA, seg_pos={0 0}, upload=473546]: Long tail cancellation due to optimal threshold Dec 28 03:27:42.967125762 (539h58m42.838208797s) [seg_pos={0 0}, upload=473546]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:27:42.967215403 (539h58m42.838298438s) [upload=473546]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:27:42.967221435 (539h58m42.83830447s) [upload=473546]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:27:43.250779947 (539h58m43.121862982s) [upload=473546]: Done uploading segments... Dec 28 03:27:43.250797010 (539h58m43.121880045s) [upload=473546]: Upload finished. err: <nil> Duration: 30.003722678s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......X........X....X...........X.....XX.....X..X.X....X.XX] duration: 5.00106259s trace: https://jaeger.dev.storj.io/trace/7f15fb8f979c932a FAIL (2025-12-28 03:28:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001090142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X...X.....] duration: 8.857783131s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:27:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 8.85780341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X...X....] duration: 52.8905688s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:27:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 52.890594861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........X........X....X...........X.....XX.....X..X.X....X.X] duration: 5.000865997s trace: https://jaeger.dev.storj.io/trace/7141932dd4dae518 FAIL (2025-12-28 03:27:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000888919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [.XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXX] duration: 30.004060069s trace: https://jaeger.dev.storj.io/trace/76c0c9bd786d56d4 FAIL (2025-12-28 03:27:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: SWIPCAFLLSVM6LFD6UZ42IOSNNIKOUFHFP7DALWYXQMOW5IZ4BNA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: SEGQVHBUUJL75WHJPP3E2Q3OHPLA7WKA4JRADA5GQBRSDFS75KDA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 4CJYNRY5LSUX6THZMXGCOACTBJHVG3DQHHEOW44ERYADDQ4SWGFQ) context deadline exceeded ======LOGS====== Dec 28 03:26:58.695719501 (539h57m58.566802536s) [upload=473538]: Starting upload Dec 28 03:26:58.695770227 (539h57m58.566853272s) [upload=473538]: Done starting upload Dec 28 03:26:58.695898211 (539h57m58.566981256s) [upload=473538]: Uploading segments... Dec 28 03:26:58.696029201 (539h57m58.567112246s) [upload=473538]: Got next segment. Inline: false Dec 28 03:26:58.696125314 (539h57m58.567208359s) [upload=473538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:26:58.696133219 (539h57m58.567216264s) [upload=473538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:26:58.946419566 (539h57m58.817502601s) [seg_pos={0 0}, upload=473538]: Begin upload segment... Dec 28 03:26:58.946500660 (539h57m58.817583705s) [seg_pos={0 0}, upload=473538]: Done begin upload segment. Dec 28 03:26:58.946611892 (539h57m58.817694927s) [noise=true , address=145.79.227.52:4446 , piece_id=TZU43ZS23E66HYN5SW5ICOJPGRMV3AIAI6TSTQ424UMFEZFWRXAA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.946629145 (539h57m58.81771219s) [noise=true , address=77.90.31.18:4808 , piece_id=5FNPVQ5DTVWSZLW3EU243GBGUISA442F5ZXQAQTPAQHMBNGXE3OA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.947126873 (539h57m58.818209908s) [noise=true , address=45.91.116.9:4261 , piece_id=4CJYNRY5LSUX6THZMXGCOACTBJHVG3DQHHEOW44ERYADDQ4SWGFQ, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.947550449 (539h57m58.818633494s) [noise=true , address=145.79.227.7:4295 , piece_id=3JLLHQKSB6Q52P5AJOI6UQ4M7XMQJQEISHPJ4UZF3C7IXE6SP55Q, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.947942576 (539h57m58.819025621s) [noise=true , address=145.79.227.47:4267 , piece_id=LRUNAU4DJLL7SI2HY4TIDJMEM6UV2LJHBKEWQTTCD2DOXRVA5EZA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.948332087 (539h57m58.819415122s) [noise=true , address=145.79.227.14:4839 , piece_id=5KRIFAGEVVJYKJQEKAX4L7NUYJOMBZTCQ4KPJNKKIURDPHY66YLA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.948757929 (539h57m58.819840964s) [noise=true , address=145.79.227.19:4077 , piece_id=VKPDUHEJNIECHHPHH67FGONAONY432MUYP2AGCFBYBNWQAMF47IQ, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.949275544 (539h57m58.820358589s) [noise=true , address=145.79.227.27:4691 , piece_id=Y5X3QJOBBNUJ43GBV2BPVWF7SYHEFONDGCFGUKKCGAGYFAS3OTRQ, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.950146422 (539h57m58.821229457s) [noise=true , address=145.79.227.43:4158 , piece_id=Z6ZMVIKSGYQKHR5FWBMRWBJFLYY5752EWUOT3H5AVC6XL25AUROA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.950819033 (539h57m58.821902068s) [noise=true , address=45.91.116.6:4974 , piece_id=DJ2QCBBSO7PHUD4B25ZTGAVYIBKIJWWPHDPRXCTYI345AGOBWLXQ, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.951307584 (539h57m58.822390619s) [noise=true , address=45.91.116.12:4225 , piece_id=SWIPCAFLLSVM6LFD6UZ42IOSNNIKOUFHFP7DALWYXQMOW5IZ4BNA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.952302207 (539h57m58.823385252s) [noise=true , address=77.90.31.21:4880 , piece_id=SEGQVHBUUJL75WHJPP3E2Q3OHPLA7WKA4JRADA5GQBRSDFS75KDA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.952393080 (539h57m58.823476115s) [noise=true , address=145.79.227.11:4730 , piece_id=PEJYW4UD6QI66ZZDW7GYKZWUDL7ZSZYB5GFTATYA533KZS3VHICA, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.953845196 (539h57m58.824928241s) [noise=true , address=77.90.31.12:4590 , piece_id=N7XVUGQBQNHM3Y6A6G2X6DYTSCTGP6ENYEKEM3DDYYXYGLL47HHQ, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.955673156 (539h57m58.826756191s) [noise=true , address=145.79.227.15:4875 , piece_id=NQU5VXMEODZFEG4BLU6J2UW4A5D6NVTJOKZCUPSGR3P5Q5F2WH3Q, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:26:58.957181128 (539h57m58.828264163s) [noise=true , address=145.79.227.48:4303 , piece_id=BP5F2HLUP6FIHG366K3CFPP5Z5Z5N4MMJANVWROYYQZ6A4SAQIXQ, seg_pos={0 0}, upload=473538]: Uploading piece... Dec 28 03:27:00.729757629 (539h58m0.600840664s) [upload=473538]: Next returned nil segment Dec 28 03:27:00.729791574 (539h58m0.600874609s) [upload=473538]: Waiting for error group managing segments... Dec 28 03:27:01.008196193 (539h58m0.879279228s) [noise=true , address=145.79.227.11:4730 , piece_id=PEJYW4UD6QI66ZZDW7GYKZWUDL7ZSZYB5GFTATYA533KZS3VHICA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.018638196 (539h58m0.889721241s) [noise=true , address=145.79.227.52:4446 , piece_id=TZU43ZS23E66HYN5SW5ICOJPGRMV3AIAI6TSTQ424UMFEZFWRXAA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.028978806 (539h58m0.900061842s) [noise=true , address=145.79.227.48:4303 , piece_id=BP5F2HLUP6FIHG366K3CFPP5Z5Z5N4MMJANVWROYYQZ6A4SAQIXQ, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.032089931 (539h58m0.903172966s) [noise=true , address=145.79.227.43:4158 , piece_id=Z6ZMVIKSGYQKHR5FWBMRWBJFLYY5752EWUOT3H5AVC6XL25AUROA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.052332035 (539h58m0.92341507s) [noise=true , address=145.79.227.14:4839 , piece_id=5KRIFAGEVVJYKJQEKAX4L7NUYJOMBZTCQ4KPJNKKIURDPHY66YLA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.059185746 (539h58m0.930268781s) [noise=true , address=145.79.227.19:4077 , piece_id=VKPDUHEJNIECHHPHH67FGONAONY432MUYP2AGCFBYBNWQAMF47IQ, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.143687722 (539h58m1.014770767s) [noise=true , address=77.90.31.12:4590 , piece_id=N7XVUGQBQNHM3Y6A6G2X6DYTSCTGP6ENYEKEM3DDYYXYGLL47HHQ, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.158271261 (539h58m1.029354307s) [noise=true , address=145.79.227.7:4295 , piece_id=3JLLHQKSB6Q52P5AJOI6UQ4M7XMQJQEISHPJ4UZF3C7IXE6SP55Q, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.246923921 (539h58m1.118006966s) [noise=true , address=77.90.31.21:4880 , piece_id=SEGQVHBUUJL75WHJPP3E2Q3OHPLA7WKA4JRADA5GQBRSDFS75KDA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.255074651 (539h58m1.126157696s) [noise=true , address=77.90.31.18:4808 , piece_id=5FNPVQ5DTVWSZLW3EU243GBGUISA442F5ZXQAQTPAQHMBNGXE3OA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.325868975 (539h58m1.19695201s) [noise=true , address=45.91.116.9:4261 , piece_id=4CJYNRY5LSUX6THZMXGCOACTBJHVG3DQHHEOW44ERYADDQ4SWGFQ, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:01.392165765 (539h58m1.26324881s) [noise=true , address=45.91.116.12:4225 , piece_id=SWIPCAFLLSVM6LFD6UZ42IOSNNIKOUFHFP7DALWYXQMOW5IZ4BNA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:02.126310750 (539h58m1.997393795s) [noise=true , address=145.79.227.15:4875 , piece_id=NQU5VXMEODZFEG4BLU6J2UW4A5D6NVTJOKZCUPSGR3P5Q5F2WH3Q, seg_pos={0 0}, upload=473538]: Done uploading piece. err: <nil> Dec 28 03:27:02.126338353 (539h58m1.997421408s) [seg_pos={0 0}, upload=473538]: Segment reached optimal threshold of 13 pieces. Dec 28 03:27:02.126884182 (539h58m1.997967227s) [noise=true , address=145.79.227.47:4267 , piece_id=LRUNAU4DJLL7SI2HY4TIDJMEM6UV2LJHBKEWQTTCD2DOXRVA5EZA, seg_pos={0 0}, upload=473538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF Dec 28 03:27:02.126894692 (539h58m1.997977737s) [noise=true , address=145.79.227.47:4267 , piece_id=LRUNAU4DJLL7SI2HY4TIDJMEM6UV2LJHBKEWQTTCD2DOXRVA5EZA, seg_pos={0 0}, upload=473538]: Long tail cancellation due to optimal threshold Dec 28 03:27:02.126926192 (539h58m1.998009267s) [noise=true , address=145.79.227.27:4691 , piece_id=Y5X3QJOBBNUJ43GBV2BPVWF7SYHEFONDGCFGUKKCGAGYFAS3OTRQ, seg_pos={0 0}, upload=473538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 28 03:27:02.126940399 (539h58m1.998023444s) [noise=true , address=145.79.227.27:4691 , piece_id=Y5X3QJOBBNUJ43GBV2BPVWF7SYHEFONDGCFGUKKCGAGYFAS3OTRQ, seg_pos={0 0}, upload=473538]: Long tail cancellation due to optimal threshold Dec 28 03:27:02.127149537 (539h58m1.998232592s) [noise=true , address=45.91.116.6:4974 , piece_id=DJ2QCBBSO7PHUD4B25ZTGAVYIBKIJWWPHDPRXCTYI345AGOBWLXQ, seg_pos={0 0}, upload=473538]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 03:27:02.127158875 (539h58m1.99824191s) [noise=true , address=45.91.116.6:4974 , piece_id=DJ2QCBBSO7PHUD4B25ZTGAVYIBKIJWWPHDPRXCTYI345AGOBWLXQ, seg_pos={0 0}, upload=473538]: Long tail cancellation due to optimal threshold Dec 28 03:27:02.127184925 (539h58m1.99826797s) [seg_pos={0 0}, upload=473538]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:27:02.127311516 (539h58m1.998394561s) [upload=473538]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:27:02.127319341 (539h58m1.998402386s) [upload=473538]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:27:02.427103820 (539h58m2.298186875s) [upload=473538]: Done uploading segments... Dec 28 03:27:02.427128508 (539h58m2.298211553s) [upload=473538]: Upload finished. err: <nil> Duration: 30.004073464s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........X........X....X...........X.....XX.....X..X.X....X.] duration: 4.982009108s trace: https://jaeger.dev.storj.io/trace/6f41be7fa1fe3f05 FAIL (2025-12-28 03:27:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.982025299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXX] duration: 30.003241395s trace: https://jaeger.dev.storj.io/trace/6306e2d738db0fc0 FAIL (2025-12-28 03:26:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: L7EVCJXJGGSWPIIBAE2LSRA3A4AXR7EO6MD2KG3OB5QIGRTD3XNA) context deadline exceeded; piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: LACXHV6TC6YOWWYJT743P2PPJNRIAVWXOY3I4K7HHWFBJJMZK3GA) context deadline exceeded ======LOGS====== 8 03:26:28.468997151 (539h57m28.340080186s) [upload=473531]: Starting upload Dec 28 03:26:28.469068697 (539h57m28.340151742s) [upload=473531]: Done starting upload Dec 28 03:26:28.469195038 (539h57m28.340278083s) [upload=473531]: Uploading segments... Dec 28 03:26:28.469383346 (539h57m28.340466411s) [upload=473531]: Got next segment. Inline: false Dec 28 03:26:28.469451767 (539h57m28.340534802s) [upload=473531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:26:28.469459922 (539h57m28.340542987s) [upload=473531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:26:28.727934090 (539h57m28.599017145s) [seg_pos={0 0}, upload=473531]: Begin upload segment... Dec 28 03:26:28.727999094 (539h57m28.599082139s) [seg_pos={0 0}, upload=473531]: Done begin upload segment. Dec 28 03:26:28.728038239 (539h57m28.599121284s) [noise=true , address=45.91.116.12:4227 , piece_id=UZYEYTX2RBPF2HSXF2VRFHCLT6KETD6CCCP5MVN52ALLRBGYQYNQ, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.731254493 (539h57m28.602337528s) [noise=true , address=77.90.31.12:4589 , piece_id=Y5XHCS3CNIHVI5DNLWAD5KILVMJK2LBBLTXXEQVFINLLS6YU65AA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.733378898 (539h57m28.604461933s) [noise=true , address=145.79.227.49:4338 , piece_id=YN7VJM4VRRG3YHO3YYXBBHFP4LPAXNU6TLDNHQRZ765NTWVHPYEQ, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.734339607 (539h57m28.605422643s) [noise=true , address=145.79.227.17:4971 , piece_id=HQDAC6GSJ7S3YMMXPAVW64QKN4MZL2GLM5BH66DDLVM43H6SAAFA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.736273691 (539h57m28.607356736s) [noise=true , address=145.79.227.5:4221 , piece_id=YQOO6FB4LYC2GRFPXAZUN3Y7JIV43X3EBR5VDHS6LMM3OIZ7GTJQ, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.737953249 (539h57m28.609036294s) [noise=true , address=77.90.31.13:4626 , piece_id=LACXHV6TC6YOWWYJT743P2PPJNRIAVWXOY3I4K7HHWFBJJMZK3GA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.738378628 (539h57m28.609461674s) [noise=true , address=145.79.227.45:4229 , piece_id=HUVT76NVV5Y3DO65WHZ4YWQC6QJH4W5Z7AM73MSULJB5PZU3DQRQ, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.742676793 (539h57m28.613759828s) [noise=true , address=77.90.31.19:4841 , piece_id=QCSVXCYMCPMK7Z4VFQ347MPC2WFXKO7GWVNHEWK3IX2B4JJR4AGA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.745075551 (539h57m28.616158596s) [noise=true , address=77.90.31.21:4880 , piece_id=WTQSJFSDFMQWS3ENE6RODUAJNJDJYZQE6CMOZ2WPTRQXNHRCU6OQ, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.746645771 (539h57m28.617728816s) [noise=true , address=145.79.227.3:4150 , piece_id=LT2BKIB4FBKZJOVMKGAMBG2LQJAH6TGHHPNLDVBT7IRU7VHTF7LQ, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.747435554 (539h57m28.618518609s) [noise=true , address=145.79.227.39:4014 , piece_id=4M5I2TEJBPGRZJWRA33EQRRYCMTP5EEKJA6RKOAI2QX7T4YNDDPA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.749331545 (539h57m28.62041458s) [noise=true , address=145.79.227.15:4876 , piece_id=JTGKUQNEZPUZWC7ZKPYP3RUB73BTLZ6SMFY2KJCLCIXORF24OZFA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.749850994 (539h57m28.620934039s) [noise=true , address=145.79.227.48:4301 , piece_id=MJGV4ZRJPQCJGOSRHATQFMQXNRJAOYTXTHPBKOTTOI2NRL2OTA3A, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.750949866 (539h57m28.622032911s) [noise=true , address=45.91.116.9:4261 , piece_id=L7EVCJXJGGSWPIIBAE2LSRA3A4AXR7EO6MD2KG3OB5QIGRTD3XNA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.752482605 (539h57m28.62356565s) [noise=true , address=45.91.116.7:4011 , piece_id=NIGBH7MKFZVCGHWBAD4DNCOIHWGWAVC653OJQHEDQVZ6F7NVVI2Q, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:28.752731188 (539h57m28.623814223s) [noise=true , address=145.79.227.24:4583 , piece_id=E2YG7IB564RXBVSB7TT5X5YF4OLUOQROMNQUHBXRXYFCZ5HJXXPA, seg_pos={0 0}, upload=473531]: Uploading piece... Dec 28 03:26:30.359860592 (539h57m30.230943627s) [upload=473531]: Next returned nil segment Dec 28 03:26:30.359877143 (539h57m30.230960188s) [upload=473531]: Waiting for error group managing segments... Dec 28 03:26:30.762307819 (539h57m30.633390854s) [noise=true , address=145.79.227.17:4971 , piece_id=HQDAC6GSJ7S3YMMXPAVW64QKN4MZL2GLM5BH66DDLVM43H6SAAFA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.763203624 (539h57m30.634286659s) [noise=true , address=145.79.227.3:4150 , piece_id=LT2BKIB4FBKZJOVMKGAMBG2LQJAH6TGHHPNLDVBT7IRU7VHTF7LQ, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.775241937 (539h57m30.646324972s) [noise=true , address=145.79.227.24:4583 , piece_id=E2YG7IB564RXBVSB7TT5X5YF4OLUOQROMNQUHBXRXYFCZ5HJXXPA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.785191993 (539h57m30.656275028s) [noise=true , address=145.79.227.48:4301 , piece_id=MJGV4ZRJPQCJGOSRHATQFMQXNRJAOYTXTHPBKOTTOI2NRL2OTA3A, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.787761055 (539h57m30.6588441s) [noise=true , address=145.79.227.39:4014 , piece_id=4M5I2TEJBPGRZJWRA33EQRRYCMTP5EEKJA6RKOAI2QX7T4YNDDPA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.808174647 (539h57m30.679257692s) [noise=true , address=145.79.227.15:4876 , piece_id=JTGKUQNEZPUZWC7ZKPYP3RUB73BTLZ6SMFY2KJCLCIXORF24OZFA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.814249173 (539h57m30.685332208s) [noise=true , address=145.79.227.45:4229 , piece_id=HUVT76NVV5Y3DO65WHZ4YWQC6QJH4W5Z7AM73MSULJB5PZU3DQRQ, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.943122631 (539h57m30.814205666s) [noise=true , address=77.90.31.19:4841 , piece_id=QCSVXCYMCPMK7Z4VFQ347MPC2WFXKO7GWVNHEWK3IX2B4JJR4AGA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:30.974115706 (539h57m30.845211996s) [noise=true , address=45.91.116.12:4227 , piece_id=UZYEYTX2RBPF2HSXF2VRFHCLT6KETD6CCCP5MVN52ALLRBGYQYNQ, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:31.002012867 (539h57m30.873095922s) [noise=true , address=77.90.31.13:4626 , piece_id=LACXHV6TC6YOWWYJT743P2PPJNRIAVWXOY3I4K7HHWFBJJMZK3GA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:31.034051123 (539h57m30.905134168s) [noise=true , address=45.91.116.9:4261 , piece_id=L7EVCJXJGGSWPIIBAE2LSRA3A4AXR7EO6MD2KG3OB5QIGRTD3XNA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:31.036137867 (539h57m30.907220912s) [noise=true , address=77.90.31.21:4880 , piece_id=WTQSJFSDFMQWS3ENE6RODUAJNJDJYZQE6CMOZ2WPTRQXNHRCU6OQ, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:31.120850654 (539h57m30.991933699s) [noise=true , address=45.91.116.7:4011 , piece_id=NIGBH7MKFZVCGHWBAD4DNCOIHWGWAVC653OJQHEDQVZ6F7NVVI2Q, seg_pos={0 0}, upload=473531]: Done uploading piece. err: <nil> Dec 28 03:26:31.120875271 (539h57m30.991958306s) [seg_pos={0 0}, upload=473531]: Segment reached optimal threshold of 13 pieces. Dec 28 03:26:31.121086313 (539h57m30.992169358s) [noise=true , address=145.79.227.49:4338 , piece_id=YN7VJM4VRRG3YHO3YYXBBHFP4LPAXNU6TLDNHQRZ765NTWVHPYEQ, seg_pos={0 0}, upload=473531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 03:26:31.121095430 (539h57m30.992178475s) [noise=true , address=145.79.227.49:4338 , piece_id=YN7VJM4VRRG3YHO3YYXBBHFP4LPAXNU6TLDNHQRZ765NTWVHPYEQ, seg_pos={0 0}, upload=473531]: Long tail cancellation due to optimal threshold Dec 28 03:26:31.121322873 (539h57m30.992405908s) [noise=true , address=77.90.31.12:4589 , piece_id=Y5XHCS3CNIHVI5DNLWAD5KILVMJK2LBBLTXXEQVFINLLS6YU65AA, seg_pos={0 0}, upload=473531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 28 03:26:31.121331700 (539h57m30.992414745s) [noise=true , address=77.90.31.12:4589 , piece_id=Y5XHCS3CNIHVI5DNLWAD5KILVMJK2LBBLTXXEQVFINLLS6YU65AA, seg_pos={0 0}, upload=473531]: Long tail cancellation due to optimal threshold Dec 28 03:26:31.121350866 (539h57m30.992434011s) [noise=true , address=145.79.227.5:4221 , piece_id=YQOO6FB4LYC2GRFPXAZUN3Y7JIV43X3EBR5VDHS6LMM3OIZ7GTJQ, seg_pos={0 0}, upload=473531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:53416->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:53416->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53416->145.79.227.5:4221: use of closed network connection Dec 28 03:26:31.121360956 (539h57m30.992443991s) [noise=true , address=145.79.227.5:4221 , piece_id=YQOO6FB4LYC2GRFPXAZUN3Y7JIV43X3EBR5VDHS6LMM3OIZ7GTJQ, seg_pos={0 0}, upload=473531]: Long tail cancellation due to optimal threshold Dec 28 03:26:31.121497356 (539h57m30.992580391s) [seg_pos={0 0}, upload=473531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:26:31.121627342 (539h57m30.992710377s) [upload=473531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:26:31.121634878 (539h57m30.992717913s) [upload=473531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:26:31.413768068 (539h57m31.284851113s) [upload=473531]: Done uploading segments... Dec 28 03:26:31.413784890 (539h57m31.284867925s) [upload=473531]: Upload finished. err: <nil> Duration: 30.003264768s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 37.10597786s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:26:33 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 37.10599353s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........X........X....X...........X.....XX.....X..X.X....X] duration: 5.001717232s trace: https://jaeger.dev.storj.io/trace/6f41be7fa1fe3f05 FAIL (2025-12-28 03:26:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001741539s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 50 out of 60 is failed (threshold=3) [X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXX] duration: 30.002689486s trace: https://jaeger.dev.storj.io/trace/2f8750eb569abcc9 FAIL (2025-12-28 03:26:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: CS4VRVAAUCBH4M56N3HGD6YZBXLVJOOCTP6PLQFISXA6AYHGX6KA) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: GHPHM3KEGK7ATQ7O55QEX7RVLI66O2C3Z5NDVND4UJJBONKFK5CQ) context deadline exceeded ======LOGS====== ... Dec 28 03:25:49.384900090 (539h56m49.255983125s) [upload=473522]: Got next segment. Inline: false Dec 28 03:25:49.384941108 (539h56m49.256024143s) [upload=473522]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:25:49.384947340 (539h56m49.256030375s) [upload=473522]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:25:49.629434003 (539h56m49.500517059s) [seg_pos={0 0}, upload=473522]: Begin upload segment... Dec 28 03:25:49.629489238 (539h56m49.500572283s) [seg_pos={0 0}, upload=473522]: Done begin upload segment. Dec 28 03:25:49.629523203 (539h56m49.500606238s) [noise=true , address=77.90.31.10:4520 , piece_id=CS4VRVAAUCBH4M56N3HGD6YZBXLVJOOCTP6PLQFISXA6AYHGX6KA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.629636529 (539h56m49.500719564s) [noise=true , address=145.79.227.27:4691 , piece_id=WNPWN2BC6VZYMCRTBOWZDT4MRLKNBZMNPL3HKYJZ6VU4BXOEMY7Q, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630156199 (539h56m49.501239244s) [noise=true , address=145.79.227.41:4085 , piece_id=LFESYW4QWJXN2X3RDETQOV7UTEQ5SGLBVBBNRJYFCMVNIIDGS3VA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630288169 (539h56m49.501371204s) [noise=true , address=45.91.116.11:4118 , piece_id=CIVKMSV7UUNANHC46UVUAET4I4Y7VU645N52NVBQCUBLKSHBQ4CQ, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630326773 (539h56m49.501409808s) [noise=true , address=145.79.227.51:4411 , piece_id=UTZE2O7C6UO2BT6TOCHGWM3OECCWLOUEQ2TZ6VSSVDIKW6APJZEQ, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630369084 (539h56m49.501452129s) [noise=true , address=145.79.227.18:4007 , piece_id=QIRNXJY2WNDHZVYKGDZ3JD33FDZP4HWBHO7FSW6RY4KGAAFI5YEA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630407918 (539h56m49.501490953s) [noise=true , address=145.79.227.49:4337 , piece_id=3BK53MN3FWVCHXBTCL6AJ7C7CTN6AFMA726BWWL4TMJ3TCACUFOA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630434489 (539h56m49.501517524s) [noise=true , address=45.91.116.7:4009 , piece_id=GHPHM3KEGK7ATQ7O55QEX7RVLI66O2C3Z5NDVND4UJJBONKFK5CQ, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630461059 (539h56m49.501544104s) [noise=true , address=45.91.116.13:4155 , piece_id=ICKRGE6EJOUZSYOL7JNSNOMAZQD2YLFMUYIV74LAUFDTZEFESSHA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.630493190 (539h56m49.501576235s) [noise=true , address=145.79.227.7:4295 , piece_id=CZDFVQKEHTMQLJ4MD5QANM7YCDT3QJ25Q5REDRABS3GEX2CXLX2Q, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.631261773 (539h56m49.502344818s) [noise=true , address=145.79.227.46:4603 , piece_id=SB3J4WCQBTG7SYT7O4L3T5W2NLX4NQBTPLAR6FXAVC3LWSFL3ZRA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.632319107 (539h56m49.503402152s) [noise=true , address=145.79.227.38:4979 , piece_id=UEELYV3QXMUQXFY4QBH3KQJT5PMVTBWMFG3PV6EMAFUEWN637V3A, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.632858744 (539h56m49.503941789s) [noise=true , address=145.79.227.15:4874 , piece_id=4GQOD6JJ2EPHPJUV5UOGC3X6OE7XBKK65LW4LNHEYUSNNXVR7SSA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.633361833 (539h56m49.504444878s) [noise=true , address=145.79.227.12:4765 , piece_id=GQGZZNNS3MXJKIN2KWUFPSYOFAJDRP5NJTRKVYPUJ3PQJCHOHVGA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.633888005 (539h56m49.50497104s) [noise=true , address=45.91.116.9:4261 , piece_id=WTGFPKYFQ5NE6DGT2NC6EAUUY2QT5KT6TMQ7D6EPNDTGQWW6UAWA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:49.634382085 (539h56m49.50546513s) [noise=true , address=145.79.227.17:4970 , piece_id=E4EHTYHCO3WXBSOMBUBZAEDFTTURWSZSEPXDLDV254ZX24AYD6NA, seg_pos={0 0}, upload=473522]: Uploading piece... Dec 28 03:25:51.255165458 (539h56m51.126248493s) [upload=473522]: Next returned nil segment Dec 28 03:25:51.255185555 (539h56m51.126268591s) [upload=473522]: Waiting for error group managing segments... Dec 28 03:25:51.642360832 (539h56m51.513443867s) [noise=true , address=145.79.227.15:4874 , piece_id=4GQOD6JJ2EPHPJUV5UOGC3X6OE7XBKK65LW4LNHEYUSNNXVR7SSA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.648455808 (539h56m51.519538854s) [noise=true , address=145.79.227.38:4979 , piece_id=UEELYV3QXMUQXFY4QBH3KQJT5PMVTBWMFG3PV6EMAFUEWN637V3A, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.652169860 (539h56m51.523252895s) [noise=true , address=145.79.227.51:4411 , piece_id=UTZE2O7C6UO2BT6TOCHGWM3OECCWLOUEQ2TZ6VSSVDIKW6APJZEQ, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.656449950 (539h56m51.527533015s) [noise=true , address=145.79.227.46:4603 , piece_id=SB3J4WCQBTG7SYT7O4L3T5W2NLX4NQBTPLAR6FXAVC3LWSFL3ZRA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.658891108 (539h56m51.529974143s) [noise=true , address=145.79.227.18:4007 , piece_id=QIRNXJY2WNDHZVYKGDZ3JD33FDZP4HWBHO7FSW6RY4KGAAFI5YEA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.675746956 (539h56m51.546829991s) [noise=true , address=145.79.227.12:4765 , piece_id=GQGZZNNS3MXJKIN2KWUFPSYOFAJDRP5NJTRKVYPUJ3PQJCHOHVGA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.676757810 (539h56m51.547840856s) [noise=true , address=145.79.227.41:4085 , piece_id=LFESYW4QWJXN2X3RDETQOV7UTEQ5SGLBVBBNRJYFCMVNIIDGS3VA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.691695975 (539h56m51.56277902s) [noise=true , address=145.79.227.7:4295 , piece_id=CZDFVQKEHTMQLJ4MD5QANM7YCDT3QJ25Q5REDRABS3GEX2CXLX2Q, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.736129534 (539h56m51.607212579s) [noise=true , address=145.79.227.27:4691 , piece_id=WNPWN2BC6VZYMCRTBOWZDT4MRLKNBZMNPL3HKYJZ6VU4BXOEMY7Q, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:51.922832457 (539h56m51.793915502s) [noise=true , address=77.90.31.10:4520 , piece_id=CS4VRVAAUCBH4M56N3HGD6YZBXLVJOOCTP6PLQFISXA6AYHGX6KA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:52.002826640 (539h56m51.873909675s) [noise=true , address=45.91.116.9:4261 , piece_id=WTGFPKYFQ5NE6DGT2NC6EAUUY2QT5KT6TMQ7D6EPNDTGQWW6UAWA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:52.060454763 (539h56m51.931537808s) [noise=true , address=45.91.116.13:4155 , piece_id=ICKRGE6EJOUZSYOL7JNSNOMAZQD2YLFMUYIV74LAUFDTZEFESSHA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:52.096782176 (539h56m51.967865221s) [noise=true , address=45.91.116.7:4009 , piece_id=GHPHM3KEGK7ATQ7O55QEX7RVLI66O2C3Z5NDVND4UJJBONKFK5CQ, seg_pos={0 0}, upload=473522]: Done uploading piece. err: <nil> Dec 28 03:25:52.096812274 (539h56m51.967895329s) [seg_pos={0 0}, upload=473522]: Segment reached optimal threshold of 13 pieces. Dec 28 03:25:52.097039777 (539h56m51.968122822s) [noise=true , address=45.91.116.11:4118 , piece_id=CIVKMSV7UUNANHC46UVUAET4I4Y7VU645N52NVBQCUBLKSHBQ4CQ, seg_pos={0 0}, upload=473522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:56052->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:56052->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56052->45.91.116.11:4118: use of closed network connection Dec 28 03:25:52.097048905 (539h56m51.96813196s) [noise=true , address=45.91.116.11:4118 , piece_id=CIVKMSV7UUNANHC46UVUAET4I4Y7VU645N52NVBQCUBLKSHBQ4CQ, seg_pos={0 0}, upload=473522]: Long tail cancellation due to optimal threshold Dec 28 03:25:52.097102296 (539h56m51.968185341s) [noise=true , address=145.79.227.17:4970 , piece_id=E4EHTYHCO3WXBSOMBUBZAEDFTTURWSZSEPXDLDV254ZX24AYD6NA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe): protocol: write tcp 188.166.225.170:54316->145.79.227.17:4970: use of closed network connection; write tcp 188.166.225.170:54316->145.79.227.17:4970: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54316->145.79.227.17:4970: use of closed network connection Dec 28 03:25:52.097142021 (539h56m51.968225056s) [noise=true , address=145.79.227.17:4970 , piece_id=E4EHTYHCO3WXBSOMBUBZAEDFTTURWSZSEPXDLDV254ZX24AYD6NA, seg_pos={0 0}, upload=473522]: Long tail cancellation due to optimal threshold Dec 28 03:25:52.097215170 (539h56m51.968298216s) [noise=true , address=145.79.227.49:4337 , piece_id=3BK53MN3FWVCHXBTCL6AJ7C7CTN6AFMA726BWWL4TMJ3TCACUFOA, seg_pos={0 0}, upload=473522]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 28 03:25:52.097223547 (539h56m51.968306602s) [noise=true , address=145.79.227.49:4337 , piece_id=3BK53MN3FWVCHXBTCL6AJ7C7CTN6AFMA726BWWL4TMJ3TCACUFOA, seg_pos={0 0}, upload=473522]: Long tail cancellation due to optimal threshold Dec 28 03:25:52.097331112 (539h56m51.968414147s) [seg_pos={0 0}, upload=473522]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:25:52.097418297 (539h56m51.968501342s) [upload=473522]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:25:52.097424379 (539h56m51.968507434s) [upload=473522]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:25:52.392191486 (539h56m52.263274521s) [upload=473522]: Done uploading segments... Dec 28 03:25:52.392215401 (539h56m52.263298447s) [upload=473522]: Upload finished. err: <nil> Duration: 30.002713331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 29.029088341s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:25:56 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 29.029115714s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...........X........X....X...........X.....XX.....X..X.X....] duration: 5.00105113s trace: https://jaeger.dev.storj.io/trace/29dde90ffce44d23 FAIL (2025-12-28 03:25:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001076698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 42.030223726s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:25:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 42.030246181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 49 out of 60 is failed (threshold=3) [.X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXX] duration: 30.003828809s trace: https://jaeger.dev.storj.io/trace/394a6063502ff662 FAIL (2025-12-28 03:25:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: EQVJQQ6JQVON5ATLYHGRZNFEW5DZHDLIMZDHXU6HL2GJOI47WZFA) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 3REP2GNJHONWJRWCRHQ7ZO35U6AVHP3FZI2CRGOD33ST2EQPM2LQ) context deadline exceeded ======LOGS====== Dec 28 03:24:46.072386904 (539h55m45.94346994s) [upload=473509]: Starting upload Dec 28 03:24:46.072456948 (539h55m45.943539983s) [upload=473509]: Done starting upload Dec 28 03:24:46.072527582 (539h55m45.943610627s) [upload=473509]: Uploading segments... Dec 28 03:24:46.072758653 (539h55m45.943841688s) [upload=473509]: Got next segment. Inline: false Dec 28 03:24:46.072820410 (539h55m45.943903446s) [upload=473509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:24:46.072839947 (539h55m45.943922982s) [upload=473509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:24:46.320725703 (539h55m46.191808749s) [seg_pos={0 0}, upload=473509]: Begin upload segment... Dec 28 03:24:46.320789094 (539h55m46.191872139s) [seg_pos={0 0}, upload=473509]: Done begin upload segment. Dec 28 03:24:46.320821857 (539h55m46.191904892s) [noise=true , address=145.79.227.19:4079 , piece_id=7HZYB7A4LJTTQMEOSMUIXOHU4LBKEBPSC6OQBEAMFDR3HMB7XUKQ, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.320907610 (539h55m46.191990645s) [noise=true , address=145.79.227.9:4365 , piece_id=NSCBZKWI43EN6KW7IHWZN534K77EULFA4GGEHVM2H4HML3SEY6IA, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.321578577 (539h55m46.192661632s) [noise=true , address=77.90.31.12:4589 , piece_id=SPZXP5TD5YAN5U5NVYDHZMNL4HLCY7KJPJHAXOYNRP5ZAQK6PIZA, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.322078239 (539h55m46.193161274s) [noise=true , address=77.90.31.21:4880 , piece_id=EQVJQQ6JQVON5ATLYHGRZNFEW5DZHDLIMZDHXU6HL2GJOI47WZFA, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.322510321 (539h55m46.193593356s) [noise=true , address=145.79.227.11:4730 , piece_id=3GVY4CXQ5S3PXUCTNDSTTS5CJK4WYPW2TU75754KPGGWKUWNAWRA, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.323129481 (539h55m46.194212516s) [noise=true , address=145.79.227.16:4909 , piece_id=FY3YPT2BFASSG3DU5VK3I7Q2IXFDIO27NC54CCC7OSRHDUOCSU4A, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.323641064 (539h55m46.194724109s) [noise=true , address=145.79.227.36:4518 , piece_id=36S7S5OOPTR5SERNMLEGG5XKSBU4TTBJ6ICMBCQIP2LSWUTFD76A, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.324161705 (539h55m46.195244741s) [noise=true , address=45.91.116.11:4118 , piece_id=EPYFHLCCFQORYEPBNXIKOYNXQ7MDNFT6UJI6Q67VDM2CI4KITP3A, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.324652521 (539h55m46.195735566s) [noise=true , address=45.91.116.4:4939 , piece_id=YB27GFLO2ECII4TOH634VCLGHXJDIOGABOZKHUMC5FTZMX7IBKPQ, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.325116865 (539h55m46.19619991s) [noise=true , address=77.90.31.13:4629 , piece_id=3REP2GNJHONWJRWCRHQ7ZO35U6AVHP3FZI2CRGOD33ST2EQPM2LQ, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.325578574 (539h55m46.196661619s) [noise=true , address=145.79.227.41:4086 , piece_id=KC4QT4PSEC2ZI53EGKUKTTRNB4CCRLPGM5Q6AYVPT24JSCQ2M2ZA, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.326184248 (539h55m46.197267303s) [noise=true , address=145.79.227.3:4151 , piece_id=F3L6YZPLTPCUHZD3SHYVCOABDJHP3BQMYL2MRB5OLZUAYVGASYGQ, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.326719106 (539h55m46.197802151s) [noise=true , address=45.91.116.5:4298 , piece_id=OLK3JHDVHNTU5ETRE76PRCG6T7U775XTPUJPRDBMJB5DJUZKRPWA, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.327213577 (539h55m46.198296622s) [noise=true , address=145.79.227.38:4979 , piece_id=OZPSXDDDQ3QMK7JUWRSFJUMJMFV5Z24RDSR6RWCG5QLC75RY364Q, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.327711555 (539h55m46.1987946s) [noise=true , address=145.79.227.13:4802 , piece_id=H6NMRZ4K7EN4UPA4ZG227A62QBDKLEG2SVZFZ7ITIXXWQUGDHJLQ, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:46.328180659 (539h55m46.199263704s) [noise=true , address=45.91.116.13:4154 , piece_id=MME2XGVQLJHAHMCI4CUQIVDCDKKWXPQSUHYQUFA3MPFKGF2JGX4Q, seg_pos={0 0}, upload=473509]: Uploading piece... Dec 28 03:24:48.071833345 (539h55m47.94291638s) [upload=473509]: Next returned nil segment Dec 28 03:24:48.071849054 (539h55m47.9429321s) [upload=473509]: Waiting for error group managing segments... Dec 28 03:24:48.360698873 (539h55m48.231781908s) [noise=true , address=145.79.227.9:4365 , piece_id=NSCBZKWI43EN6KW7IHWZN534K77EULFA4GGEHVM2H4HML3SEY6IA, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.364932224 (539h55m48.236015259s) [noise=true , address=145.79.227.13:4802 , piece_id=H6NMRZ4K7EN4UPA4ZG227A62QBDKLEG2SVZFZ7ITIXXWQUGDHJLQ, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.372669727 (539h55m48.243752772s) [noise=true , address=145.79.227.41:4086 , piece_id=KC4QT4PSEC2ZI53EGKUKTTRNB4CCRLPGM5Q6AYVPT24JSCQ2M2ZA, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.418723131 (539h55m48.289806176s) [noise=true , address=145.79.227.19:4079 , piece_id=7HZYB7A4LJTTQMEOSMUIXOHU4LBKEBPSC6OQBEAMFDR3HMB7XUKQ, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.419789452 (539h55m48.290872487s) [noise=true , address=145.79.227.11:4730 , piece_id=3GVY4CXQ5S3PXUCTNDSTTS5CJK4WYPW2TU75754KPGGWKUWNAWRA, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.445241276 (539h55m48.316324311s) [noise=true , address=145.79.227.16:4909 , piece_id=FY3YPT2BFASSG3DU5VK3I7Q2IXFDIO27NC54CCC7OSRHDUOCSU4A, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.447084917 (539h55m48.318167952s) [noise=true , address=145.79.227.36:4518 , piece_id=36S7S5OOPTR5SERNMLEGG5XKSBU4TTBJ6ICMBCQIP2LSWUTFD76A, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.517590822 (539h55m48.388673857s) [noise=true , address=77.90.31.13:4629 , piece_id=3REP2GNJHONWJRWCRHQ7ZO35U6AVHP3FZI2CRGOD33ST2EQPM2LQ, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:48.686833868 (539h55m48.557916913s) [noise=true , address=45.91.116.13:4154 , piece_id=MME2XGVQLJHAHMCI4CUQIVDCDKKWXPQSUHYQUFA3MPFKGF2JGX4Q, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:49.007109827 (539h55m48.878192863s) [noise=true , address=77.90.31.21:4880 , piece_id=EQVJQQ6JQVON5ATLYHGRZNFEW5DZHDLIMZDHXU6HL2GJOI47WZFA, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:49.188554559 (539h55m49.059637594s) [noise=true , address=145.79.227.3:4151 , piece_id=F3L6YZPLTPCUHZD3SHYVCOABDJHP3BQMYL2MRB5OLZUAYVGASYGQ, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:49.249640236 (539h55m49.120723271s) [noise=true , address=45.91.116.4:4939 , piece_id=YB27GFLO2ECII4TOH634VCLGHXJDIOGABOZKHUMC5FTZMX7IBKPQ, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:49.270470390 (539h55m49.141553435s) [noise=true , address=45.91.116.5:4298 , piece_id=OLK3JHDVHNTU5ETRE76PRCG6T7U775XTPUJPRDBMJB5DJUZKRPWA, seg_pos={0 0}, upload=473509]: Done uploading piece. err: <nil> Dec 28 03:24:49.270495919 (539h55m49.141578964s) [seg_pos={0 0}, upload=473509]: Segment reached optimal threshold of 13 pieces. Dec 28 03:24:49.270838221 (539h55m49.141921256s) [noise=true , address=145.79.227.38:4979 , piece_id=OZPSXDDDQ3QMK7JUWRSFJUMJMFV5Z24RDSR6RWCG5QLC75RY364Q, seg_pos={0 0}, upload=473509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: expected piece hash; context canceled; EOF Dec 28 03:24:49.270846627 (539h55m49.141929662s) [noise=true , address=145.79.227.38:4979 , piece_id=OZPSXDDDQ3QMK7JUWRSFJUMJMFV5Z24RDSR6RWCG5QLC75RY364Q, seg_pos={0 0}, upload=473509]: Long tail cancellation due to optimal threshold Dec 28 03:24:49.270854030 (539h55m49.141937096s) [noise=true , address=45.91.116.11:4118 , piece_id=EPYFHLCCFQORYEPBNXIKOYNXQ7MDNFT6UJI6Q67VDM2CI4KITP3A, seg_pos={0 0}, upload=473509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 03:24:49.270865152 (539h55m49.141948187s) [noise=true , address=45.91.116.11:4118 , piece_id=EPYFHLCCFQORYEPBNXIKOYNXQ7MDNFT6UJI6Q67VDM2CI4KITP3A, seg_pos={0 0}, upload=473509]: Long tail cancellation due to optimal threshold Dec 28 03:24:49.271113515 (539h55m49.14219656s) [noise=true , address=77.90.31.12:4589 , piece_id=SPZXP5TD5YAN5U5NVYDHZMNL4HLCY7KJPJHAXOYNRP5ZAQK6PIZA, seg_pos={0 0}, upload=473509]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 28 03:24:49.271124385 (539h55m49.142207421s) [noise=true , address=77.90.31.12:4589 , piece_id=SPZXP5TD5YAN5U5NVYDHZMNL4HLCY7KJPJHAXOYNRP5ZAQK6PIZA, seg_pos={0 0}, upload=473509]: Long tail cancellation due to optimal threshold Dec 28 03:24:49.271149524 (539h55m49.14223305s) [seg_pos={0 0}, upload=473509]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:24:49.271279671 (539h55m49.142362716s) [upload=473509]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:24:49.271287316 (539h55m49.142370361s) [upload=473509]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:24:49.564483200 (539h55m49.435566235s) [upload=473509]: Done uploading segments... Dec 28 03:24:49.564503719 (539h55m49.435586754s) [upload=473509]: Upload finished. err: <nil> Duration: 30.003852374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXX] duration: 30.002628795s trace: https://jaeger.dev.storj.io/trace/784056f0b6467c69 FAIL (2025-12-28 03:24:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:24:13.733314439 (539h55m13.604397484s) [upload=473500]: Starting upload Dec 28 03:24:13.733403097 (539h55m13.604486142s) [upload=473500]: Done starting upload Dec 28 03:24:13.733475496 (539h55m13.604558541s) [upload=473500]: Uploading segments... Dec 28 03:24:13.733691256 (539h55m13.604774291s) [upload=473500]: Got next segment. Inline: false Dec 28 03:24:13.733726233 (539h55m13.604809268s) [upload=473500]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:24:13.733730992 (539h55m13.604814037s) [upload=473500]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:24:13.980777857 (539h55m13.851860902s) [seg_pos={0 0}, upload=473500]: Begin upload segment... Dec 28 03:24:13.980843873 (539h55m13.851926918s) [seg_pos={0 0}, upload=473500]: Done begin upload segment. Dec 28 03:24:13.980894549 (539h55m13.851977584s) [noise=true , address=45.91.116.7:4012 , piece_id=7MGCRX2YYDAENRVVXOQVMQ7CG47GER33ZKD5Y4LGBZRPMXPD3UKA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.980902946 (539h55m13.851985991s) [noise=true , address=77.90.31.17:4773 , piece_id=GZQ46Y6LHPN3TKL3GWYQUFE66A5IJI4WAMLIGXIAU5XA5KXBO4YA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.981210541 (539h55m13.852293576s) [noise=true , address=77.90.31.16:4736 , piece_id=QOW43TZIBUGZ6UFWVYSEVBBT2ABW2SGUKYSPBRBON5YNJ5D3UK7A, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.981513928 (539h55m13.852596963s) [noise=true , address=45.91.116.5:4300 , piece_id=KVH3YCPVC6IVGWYEO7SWH36CXT4CB3Z77SPZ7ZWLRQ3QOPFMH4RA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.981827235 (539h55m13.85291027s) [noise=true , address=145.79.227.36:4519 , piece_id=QGAFNU3BO7ON2ES56CR2FXJUILSJYAQYZOVLITJGIS3F6SAJKOMA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.982120223 (539h55m13.853203269s) [noise=true , address=145.79.227.37:4566 , piece_id=UST25C6M4BUUHCDFOGVNCQXKEZ2A4WRHBWMUX7W4K6QHOFCYSOYA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.982410075 (539h55m13.85349312s) [noise=true , address=145.79.227.44:4195 , piece_id=FJL3S34WYAKYSOFZOAOW4AJW3GQ22XCBI5DHTFM6FNUST6UNEIKQ, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.982421817 (539h55m13.853504852s) [noise=true , address=145.79.227.40:4050 , piece_id=QOGBVZ7JHKKWDJSN7EOJL7OLKUHHEI6AUR7JAX6H73MUIJSN5HRA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.982778948 (539h55m13.853861983s) [noise=true , address=145.79.227.52:4447 , piece_id=MH2LCFDDERK2DLIDDZTBWOSVHLLZYALAEHR33Z7RGSZFLBQLJTXA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.982931077 (539h55m13.854014122s) [noise=true , address=145.79.227.16:4912 , piece_id=LM4QZ7Q7OIWZDA4AJJR2M2P7SZ2JQFMLXHLXUIVXZ3AW5GXGP3ZA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.983321731 (539h55m13.854404766s) [noise=true , address=145.79.227.11:4730 , piece_id=ZJYDW4JZ5WOV7UQKUBPTHYV3C3E6V2MQCISTELSRAXFLVBHEZDEQ, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.983785754 (539h55m13.854868789s) [noise=true , address=145.79.227.39:4013 , piece_id=DXUEU6RDJ6PRGRVTHKX3UPSRHJGJABMDZZZL2EXJGBVYSETVCWGQ, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.983932885 (539h55m13.85501592s) [noise=true , address=145.79.227.32:4836 , piece_id=OV7R673T7ILWN55Q2DNQJIHFQ6YOJNGBCCD5JNGOSSZOGQKL2BRA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.984285176 (539h55m13.855368211s) [noise=true , address=45.91.116.6:4974 , piece_id=GS6I7F52AKLWWFXRFHB53UX3TBEFPNTIGT4R6VSULHDVCEJFSIYA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.984631465 (539h55m13.85571451s) [noise=true , address=145.79.227.38:4977 , piece_id=EXHEAC27HKJYQ63YU3V445CRS5S47W3Z3FNFFDP6YXFF2UFC5DQQ, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:13.985003343 (539h55m13.856086388s) [noise=true , address=145.79.227.45:4230 , piece_id=XH6UKCPDNZVAW5AUQFELC2BLRMGUSB4MX7A7UODV7WXYSF3NUOCA, seg_pos={0 0}, upload=473500]: Uploading piece... Dec 28 03:24:15.663331537 (539h55m15.534414582s) [upload=473500]: Next returned nil segment Dec 28 03:24:15.663348158 (539h55m15.534431204s) [upload=473500]: Waiting for error group managing segments... Dec 28 03:24:15.987747455 (539h55m15.85883049s) [noise=true , address=145.79.227.11:4730 , piece_id=ZJYDW4JZ5WOV7UQKUBPTHYV3C3E6V2MQCISTELSRAXFLVBHEZDEQ, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:15.990236704 (539h55m15.861319749s) [noise=true , address=145.79.227.52:4447 , piece_id=MH2LCFDDERK2DLIDDZTBWOSVHLLZYALAEHR33Z7RGSZFLBQLJTXA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.001468752 (539h55m15.872551807s) [noise=true , address=145.79.227.36:4519 , piece_id=QGAFNU3BO7ON2ES56CR2FXJUILSJYAQYZOVLITJGIS3F6SAJKOMA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.002267764 (539h55m15.873350809s) [noise=true , address=145.79.227.45:4230 , piece_id=XH6UKCPDNZVAW5AUQFELC2BLRMGUSB4MX7A7UODV7WXYSF3NUOCA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.014991812 (539h55m15.886074857s) [noise=true , address=145.79.227.38:4977 , piece_id=EXHEAC27HKJYQ63YU3V445CRS5S47W3Z3FNFFDP6YXFF2UFC5DQQ, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.018907979 (539h55m15.889991014s) [noise=true , address=145.79.227.32:4836 , piece_id=OV7R673T7ILWN55Q2DNQJIHFQ6YOJNGBCCD5JNGOSSZOGQKL2BRA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.024115014 (539h55m15.895198059s) [noise=true , address=145.79.227.16:4912 , piece_id=LM4QZ7Q7OIWZDA4AJJR2M2P7SZ2JQFMLXHLXUIVXZ3AW5GXGP3ZA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.031907242 (539h55m15.902990277s) [noise=true , address=145.79.227.39:4013 , piece_id=DXUEU6RDJ6PRGRVTHKX3UPSRHJGJABMDZZZL2EXJGBVYSETVCWGQ, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.119276577 (539h55m15.990359612s) [noise=true , address=145.79.227.37:4566 , piece_id=UST25C6M4BUUHCDFOGVNCQXKEZ2A4WRHBWMUX7W4K6QHOFCYSOYA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.163664838 (539h55m16.034747873s) [noise=true , address=77.90.31.17:4773 , piece_id=GZQ46Y6LHPN3TKL3GWYQUFE66A5IJI4WAMLIGXIAU5XA5KXBO4YA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.287650736 (539h55m16.158733781s) [noise=true , address=45.91.116.5:4300 , piece_id=KVH3YCPVC6IVGWYEO7SWH36CXT4CB3Z77SPZ7ZWLRQ3QOPFMH4RA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.349068035 (539h55m16.22015107s) [noise=true , address=45.91.116.6:4974 , piece_id=GS6I7F52AKLWWFXRFHB53UX3TBEFPNTIGT4R6VSULHDVCEJFSIYA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.468522735 (539h55m16.33960577s) [noise=true , address=45.91.116.7:4012 , piece_id=7MGCRX2YYDAENRVVXOQVMQ7CG47GER33ZKD5Y4LGBZRPMXPD3UKA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: <nil> Dec 28 03:24:16.468541710 (539h55m16.339624746s) [seg_pos={0 0}, upload=473500]: Segment reached optimal threshold of 13 pieces. Dec 28 03:24:16.468718557 (539h55m16.339801592s) [noise=true , address=77.90.31.16:4736 , piece_id=QOW43TZIBUGZ6UFWVYSEVBBT2ABW2SGUKYSPBRBON5YNJ5D3UK7A, seg_pos={0 0}, upload=473500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:57400->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:57400->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57400->77.90.31.16:4736: use of closed network connection Dec 28 03:24:16.468724800 (539h55m16.339807845s) [noise=true , address=77.90.31.16:4736 , piece_id=QOW43TZIBUGZ6UFWVYSEVBBT2ABW2SGUKYSPBRBON5YNJ5D3UK7A, seg_pos={0 0}, upload=473500]: Long tail cancellation due to optimal threshold Dec 28 03:24:16.469008480 (539h55m16.340091515s) [noise=true , address=145.79.227.44:4195 , piece_id=FJL3S34WYAKYSOFZOAOW4AJW3GQ22XCBI5DHTFM6FNUST6UNEIKQ, seg_pos={0 0}, upload=473500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 03:24:16.469014732 (539h55m16.340097777s) [noise=true , address=145.79.227.44:4195 , piece_id=FJL3S34WYAKYSOFZOAOW4AJW3GQ22XCBI5DHTFM6FNUST6UNEIKQ, seg_pos={0 0}, upload=473500]: Long tail cancellation due to optimal threshold Dec 28 03:24:16.469119291 (539h55m16.340202326s) [noise=true , address=145.79.227.40:4050 , piece_id=QOGBVZ7JHKKWDJSN7EOJL7OLKUHHEI6AUR7JAX6H73MUIJSN5HRA, seg_pos={0 0}, upload=473500]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 28 03:24:16.469152694 (539h55m16.340235739s) [noise=true , address=145.79.227.40:4050 , piece_id=QOGBVZ7JHKKWDJSN7EOJL7OLKUHHEI6AUR7JAX6H73MUIJSN5HRA, seg_pos={0 0}, upload=473500]: Long tail cancellation due to optimal threshold Dec 28 03:24:16.469169245 (539h55m16.34025228s) [seg_pos={0 0}, upload=473500]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:24:16.469248235 (539h55m16.340331271s) [upload=473500]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:24:16.469253256 (539h55m16.340336291s) [upload=473500]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:24:16.765300464 (539h55m16.636383499s) [upload=473500]: Done uploading segments... Dec 28 03:24:16.765319281 (539h55m16.636402316s) [upload=473500]: Upload finished. err: <nil> Duration: 30.002639446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.002086886s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:24:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 1m0.002099912s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [............X........X....X...........X.....XX.....X..X.X...] duration: 3.932033822s trace: https://jaeger.dev.storj.io/trace/29dde90ffce44d23 FAIL (2025-12-28 03:24:32 UTC) orbiter warning: context deadline exceeded Duration: 3.932059833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 48 out of 60 is failed (threshold=3) [X..X..XXXXX.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXX] duration: 30.003323798s trace: https://jaeger.dev.storj.io/trace/44101f4efb09996b FAIL (2025-12-28 03:24:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: IDTRN2P3P565G6M6YDGU3NWRYPR3NEZZ6RSFHTAESWBZJV2YWCIA) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 3M6EKXNCMPRSLXI55LHFZ4SSE6AVKZZGHFIYWIPOHQLOFPRXALSA) context deadline exceeded ======LOGS====== Dec 28 03:23:43.498264298 (539h54m43.369347333s) [upload=473492]: Starting upload Dec 28 03:23:43.498326896 (539h54m43.369409941s) [upload=473492]: Done starting upload Dec 28 03:23:43.498366552 (539h54m43.369449598s) [upload=473492]: Uploading segments... Dec 28 03:23:43.498623151 (539h54m43.369706196s) [upload=473492]: Got next segment. Inline: false Dec 28 03:23:43.498670160 (539h54m43.369753205s) [upload=473492]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:23:43.498676132 (539h54m43.369759167s) [upload=473492]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:23:43.742195459 (539h54m43.613278494s) [seg_pos={0 0}, upload=473492]: Begin upload segment... Dec 28 03:23:43.742242769 (539h54m43.613325804s) [seg_pos={0 0}, upload=473492]: Done begin upload segment. Dec 28 03:23:43.742289699 (539h54m43.613372734s) [noise=true , address=145.79.227.52:4447 , piece_id=O4YLOHCYE474AB2MX5LY2CXPNK4BOZU2YDR5AZUVXVPWTOTPO6BA, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.742325668 (539h54m43.613408713s) [noise=true , address=45.91.116.13:4153 , piece_id=QJCCTHK5XJPSUU2K3KGHREHQXJ6BUOELZBVB6RDDLT65K3SSP3KA, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.742402033 (539h54m43.613485068s) [noise=true , address=145.79.227.37:4566 , piece_id=Q6RP23RDXKGZW6BH3GMYMUHL2ZBNZH2ONB5YZMKSST4XRWKFHI2A, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.742969944 (539h54m43.614052989s) [noise=true , address=77.90.31.11:4556 , piece_id=6BAKBOVXJINHR6W4EDFKS3TQPHKF677HDKXSKAANEVN4PP6XJ3IQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.743272951 (539h54m43.614355986s) [noise=true , address=77.90.31.14:4664 , piece_id=3M6EKXNCMPRSLXI55LHFZ4SSE6AVKZZGHFIYWIPOHQLOFPRXALSA, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.743284443 (539h54m43.614367478s) [noise=true , address=145.79.227.4:4187 , piece_id=UN2EG263BCVZKFSTSG4AWOPGHUPQJMFMDGMYDWR4ACR2NVLZ2DGQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.743535050 (539h54m43.614618085s) [noise=true , address=145.79.227.45:4230 , piece_id=UJI7QJ3BFV5UMG3OSBVCAI672QTBIWG5LCZBVWPTPLTLJ2L6IO3Q, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.743651783 (539h54m43.614734818s) [noise=true , address=145.79.227.16:4910 , piece_id=34F5NU2ICOQT3AZEIVOXML24QEEX5DSKJVCYUIWZFBIMZB6CBGNQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.743803030 (539h54m43.614886065s) [noise=true , address=145.79.227.17:4972 , piece_id=OM4GVP477SFWP5CXDKZ4VCWRAETGKA3R5INGGJFIPHVAE6PHWQ2Q, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.744011207 (539h54m43.615094242s) [noise=true , address=145.79.227.31:4798 , piece_id=XRM7BVRFS57HKGQCSJDA54AQTNIDAHV7VVCIRDNQNTOCTYACCX2Q, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.744146915 (539h54m43.61522995s) [noise=true , address=45.91.116.8:4047 , piece_id=PI7QIPP6N6OUHIZBVUICG6DYJNCMO4RPB5B6A5TYVASPTEFR4G7Q, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.744341024 (539h54m43.615424059s) [noise=true , address=45.91.116.5:4297 , piece_id=EF2JZN7UQHG3KU3JNKVWFDLQDSNUBDZJNLU6ZNM4XIF4REM3HCGQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.746395446 (539h54m43.617478481s) [noise=true , address=145.79.227.51:4409 , piece_id=JSCM5G4TQYEV3N6B6DMWH7WD463NH2UNJLUKNSH5R5S7SCJRM3QQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.746568045 (539h54m43.61765109s) [noise=true , address=45.91.116.6:4976 , piece_id=IDTRN2P3P565G6M6YDGU3NWRYPR3NEZZ6RSFHTAESWBZJV2YWCIA, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.746926208 (539h54m43.618009283s) [noise=true , address=145.79.227.11:4729 , piece_id=EIBZPV5CRDDTUUFFGBFMAC4T7D4SU2E4UCBNDBXX25M646ICMVPQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:43.747411923 (539h54m43.618494958s) [noise=true , address=145.79.227.36:4518 , piece_id=EJZQD355AVN4BGYG3VBKNNNHL5M735NLWDDMENS4KFADL5UKOZSQ, seg_pos={0 0}, upload=473492]: Uploading piece... Dec 28 03:23:45.385705734 (539h54m45.256788769s) [upload=473492]: Next returned nil segment Dec 28 03:23:45.385720944 (539h54m45.256803979s) [upload=473492]: Waiting for error group managing segments... Dec 28 03:23:45.735835665 (539h54m45.6069187s) [noise=true , address=145.79.227.37:4566 , piece_id=Q6RP23RDXKGZW6BH3GMYMUHL2ZBNZH2ONB5YZMKSST4XRWKFHI2A, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.743671075 (539h54m45.61475411s) [noise=true , address=145.79.227.45:4230 , piece_id=UJI7QJ3BFV5UMG3OSBVCAI672QTBIWG5LCZBVWPTPLTLJ2L6IO3Q, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.751460237 (539h54m45.622543272s) [noise=true , address=145.79.227.31:4798 , piece_id=XRM7BVRFS57HKGQCSJDA54AQTNIDAHV7VVCIRDNQNTOCTYACCX2Q, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.753535297 (539h54m45.624618332s) [noise=true , address=145.79.227.16:4910 , piece_id=34F5NU2ICOQT3AZEIVOXML24QEEX5DSKJVCYUIWZFBIMZB6CBGNQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.755424375 (539h54m45.62650741s) [noise=true , address=145.79.227.52:4447 , piece_id=O4YLOHCYE474AB2MX5LY2CXPNK4BOZU2YDR5AZUVXVPWTOTPO6BA, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.756090163 (539h54m45.627173208s) [noise=true , address=145.79.227.36:4518 , piece_id=EJZQD355AVN4BGYG3VBKNNNHL5M735NLWDDMENS4KFADL5UKOZSQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.769375029 (539h54m45.640458074s) [noise=true , address=145.79.227.11:4729 , piece_id=EIBZPV5CRDDTUUFFGBFMAC4T7D4SU2E4UCBNDBXX25M646ICMVPQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.777140616 (539h54m45.648223651s) [noise=true , address=145.79.227.4:4187 , piece_id=UN2EG263BCVZKFSTSG4AWOPGHUPQJMFMDGMYDWR4ACR2NVLZ2DGQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.782030427 (539h54m45.653113472s) [noise=true , address=145.79.227.51:4409 , piece_id=JSCM5G4TQYEV3N6B6DMWH7WD463NH2UNJLUKNSH5R5S7SCJRM3QQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:45.805424843 (539h54m45.676507879s) [noise=true , address=145.79.227.17:4972 , piece_id=OM4GVP477SFWP5CXDKZ4VCWRAETGKA3R5INGGJFIPHVAE6PHWQ2Q, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:46.019562224 (539h54m45.890645269s) [noise=true , address=77.90.31.14:4664 , piece_id=3M6EKXNCMPRSLXI55LHFZ4SSE6AVKZZGHFIYWIPOHQLOFPRXALSA, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:46.042864555 (539h54m45.91394759s) [noise=true , address=77.90.31.11:4556 , piece_id=6BAKBOVXJINHR6W4EDFKS3TQPHKF677HDKXSKAANEVN4PP6XJ3IQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:46.161905597 (539h54m46.032988632s) [noise=true , address=45.91.116.6:4976 , piece_id=IDTRN2P3P565G6M6YDGU3NWRYPR3NEZZ6RSFHTAESWBZJV2YWCIA, seg_pos={0 0}, upload=473492]: Done uploading piece. err: <nil> Dec 28 03:23:46.161924794 (539h54m46.033007839s) [seg_pos={0 0}, upload=473492]: Segment reached optimal threshold of 13 pieces. Dec 28 03:23:46.162118051 (539h54m46.033201086s) [noise=true , address=45.91.116.8:4047 , piece_id=PI7QIPP6N6OUHIZBVUICG6DYJNCMO4RPB5B6A5TYVASPTEFR4G7Q, seg_pos={0 0}, upload=473492]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:46.162124453 (539h54m46.033207498s) [noise=true , address=45.91.116.8:4047 , piece_id=PI7QIPP6N6OUHIZBVUICG6DYJNCMO4RPB5B6A5TYVASPTEFR4G7Q, seg_pos={0 0}, upload=473492]: Long tail cancellation due to optimal threshold Dec 28 03:23:46.162333371 (539h54m46.033416406s) [noise=true , address=45.91.116.13:4153 , piece_id=QJCCTHK5XJPSUU2K3KGHREHQXJ6BUOELZBVB6RDDLT65K3SSP3KA, seg_pos={0 0}, upload=473492]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:46.162340475 (539h54m46.033423521s) [noise=true , address=45.91.116.13:4153 , piece_id=QJCCTHK5XJPSUU2K3KGHREHQXJ6BUOELZBVB6RDDLT65K3SSP3KA, seg_pos={0 0}, upload=473492]: Long tail cancellation due to optimal threshold Dec 28 03:23:46.162599969 (539h54m46.033683024s) [noise=true , address=45.91.116.5:4297 , piece_id=EF2JZN7UQHG3KU3JNKVWFDLQDSNUBDZJNLU6ZNM4XIF4REM3HCGQ, seg_pos={0 0}, upload=473492]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:46.162620579 (539h54m46.033703614s) [noise=true , address=45.91.116.5:4297 , piece_id=EF2JZN7UQHG3KU3JNKVWFDLQDSNUBDZJNLU6ZNM4XIF4REM3HCGQ, seg_pos={0 0}, upload=473492]: Long tail cancellation due to optimal threshold Dec 28 03:23:46.162654934 (539h54m46.033737969s) [seg_pos={0 0}, upload=473492]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:23:46.162756788 (539h54m46.033839833s) [upload=473492]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:46.162764803 (539h54m46.033847848s) [upload=473492]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:46.463435517 (539h54m46.334518552s) [upload=473492]: Done uploading segments... Dec 28 03:23:46.463454052 (539h54m46.334537087s) [upload=473492]: Upload finished. err: <nil> Duration: 30.003336292s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X............X........X....X...........X.....XX.....X..X.X..] duration: 3.505853749s trace: https://jaeger.dev.storj.io/trace/29dde90ffce44d23 FAIL (2025-12-28 03:24:07 UTC) orbiter warning: context deadline exceeded Duration: 3.50587521s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)