Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXXX] duration: 30.002477672s trace: https://jaeger.dev.storj.io/trace/1d23a38a2ce8f300 FAIL (2025-12-28 03:44:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: W6QNHQJV4RBIQFBGJEJX7GP67NCSE2DRGSXPVK6PA36MJCWONEHA) context deadline exceeded; piecestore: (Node ID: 1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, Piece ID: JYYLHQCDC7JCD3OM3LJZFHNDARXLF3LIV2GVMKUNV2MWC6USUV2Q) context deadline exceeded ======LOGS====== Dec 28 03:44:09.923143173 (540h15m9.794226218s) [upload=473797]: Starting upload Dec 28 03:44:09.923234497 (540h15m9.794317552s) [upload=473797]: Done starting upload Dec 28 03:44:09.923388009 (540h15m9.794471054s) [upload=473797]: Uploading segments... Dec 28 03:44:09.923482029 (540h15m9.794565064s) [upload=473797]: Got next segment. Inline: false Dec 28 03:44:09.923528857 (540h15m9.794611892s) [upload=473797]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:44:09.923537283 (540h15m9.794620328s) [upload=473797]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:44:10.190034390 (540h15m10.061117425s) [seg_pos={0 0}, upload=473797]: Begin upload segment... Dec 28 03:44:10.190078805 (540h15m10.06116184s) [seg_pos={0 0}, upload=473797]: Done begin upload segment. Dec 28 03:44:10.190103842 (540h15m10.061186887s) [noise=true , address=145.79.227.15:4875 , piece_id=UOG44V7TVJEA3B2TGVQYMX3A23THFVPHGAO75UTRK5IAEPLFY4MQ, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.190164799 (540h15m10.061247844s) [noise=true , address=45.91.116.8:4048 , piece_id=3RB3HZLKQXH47OJ4ZYRCGNYFVIFNZVNPIPVR4O5QNW2MUJZ33IKA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.190709346 (540h15m10.061792391s) [noise=true , address=45.91.116.6:4973 , piece_id=DBXF5T7HZGGIOQI64SPLSLOOQNN6PJD45BBQX5FEG3ITTJXWSVEA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.191042931 (540h15m10.062125966s) [noise=true , address=145.79.227.37:4567 , piece_id=UQZKYWRCKMM25EONS5GMAQRKA3TD66KJPK6X3UOW5SEWU4H5JDGQ, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.191316331 (540h15m10.062399366s) [noise=true , address=145.79.227.46:4603 , piece_id=IDUY6ZWZ73EFN56ABTPLAWG7J7467SECTTCC43HGOQCDDO3MGTPA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.191641571 (540h15m10.062724606s) [noise=true , address=45.91.116.7:4010 , piece_id=PCGK7VML6G23UPAGRB73ARAKWKLIN2OMYAKHHGA4QUWO24JATGMA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.191921614 (540h15m10.06300466s) [noise=true , address=145.79.227.17:4970 , piece_id=DVOFNP6GB6OY6KU5IDVDPJIM6XE3PCPHGGZ6ATIQMYTULWXCHV4A, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.192260810 (540h15m10.063343855s) [noise=true , address=145.79.227.47:4266 , piece_id=CB6VAAWIDDNHZJZ2GJQJUVFESIMPZRZ5B36HIU2CWYFDHAPI5TKQ, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.192596559 (540h15m10.063679594s) [noise=true , address=77.90.31.13:4629 , piece_id=3JPQRVWSK6UAODOAUU5VYGUF33ZN6KCQEFCZLZVDTZJ7IOLMW3XQ, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.193534966 (540h15m10.064618011s) [noise=true , address=45.91.116.13:4153 , piece_id=W6QNHQJV4RBIQFBGJEJX7GP67NCSE2DRGSXPVK6PA36MJCWONEHA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.194676009 (540h15m10.065759044s) [noise=true , address=145.79.227.19:4079 , piece_id=EWSE4UNKUG6VJ5OFDCI7BXRYYFEKHMZU2RUGO3VC66HIAKJ2KFYA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.196183309 (540h15m10.067266354s) [noise=true , address=145.79.227.16:4912 , piece_id=JYYLHQCDC7JCD3OM3LJZFHNDARXLF3LIV2GVMKUNV2MWC6USUV2Q, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.199095455 (540h15m10.0701785s) [noise=true , address=145.79.227.39:4014 , piece_id=WLR7UYS5KL2PX7OPDLX7L7MFMCK6EF2JOWETAYUVP2D66NN33HNA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.202652367 (540h15m10.073735402s) [noise=true , address=77.90.31.14:4663 , piece_id=4G7NBQFYRVSFPB7YWU3HQWKARO5OFGIR5HQZ4T3Y5J36WFKLEXOA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.204597121 (540h15m10.075680156s) [noise=true , address=145.79.227.11:4730 , piece_id=6K3VIB5N43C5U2G5AUVLBVBS55HLKPJFBM2EZGJCJ3ANVT6QB4MA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:10.205589169 (540h15m10.076672204s) [noise=true , address=77.90.31.12:4590 , piece_id=STSUMBGCMWREK7MHCR3TYPYGT2QPWBJCMPHDGFBRSAECUJ2FFWRA, seg_pos={0 0}, upload=473797]: Uploading piece... Dec 28 03:44:12.043731768 (540h15m11.914814813s) [upload=473797]: Next returned nil segment Dec 28 03:44:12.043748050 (540h15m11.914831095s) [upload=473797]: Waiting for error group managing segments... Dec 28 03:44:12.349483362 (540h15m12.220566397s) [noise=true , address=145.79.227.17:4970 , piece_id=DVOFNP6GB6OY6KU5IDVDPJIM6XE3PCPHGGZ6ATIQMYTULWXCHV4A, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.356365136 (540h15m12.227448181s) [noise=true , address=145.79.227.37:4567 , piece_id=UQZKYWRCKMM25EONS5GMAQRKA3TD66KJPK6X3UOW5SEWU4H5JDGQ, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.357573186 (540h15m12.228656241s) [noise=true , address=145.79.227.11:4730 , piece_id=6K3VIB5N43C5U2G5AUVLBVBS55HLKPJFBM2EZGJCJ3ANVT6QB4MA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.363219077 (540h15m12.234302122s) [noise=true , address=145.79.227.46:4603 , piece_id=IDUY6ZWZ73EFN56ABTPLAWG7J7467SECTTCC43HGOQCDDO3MGTPA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.371471501 (540h15m12.242554546s) [noise=true , address=145.79.227.15:4875 , piece_id=UOG44V7TVJEA3B2TGVQYMX3A23THFVPHGAO75UTRK5IAEPLFY4MQ, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.380850159 (540h15m12.251933205s) [noise=true , address=145.79.227.19:4079 , piece_id=EWSE4UNKUG6VJ5OFDCI7BXRYYFEKHMZU2RUGO3VC66HIAKJ2KFYA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.391644343 (540h15m12.262727388s) [noise=true , address=145.79.227.47:4266 , piece_id=CB6VAAWIDDNHZJZ2GJQJUVFESIMPZRZ5B36HIU2CWYFDHAPI5TKQ, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.439020756 (540h15m12.310103801s) [noise=true , address=145.79.227.16:4912 , piece_id=JYYLHQCDC7JCD3OM3LJZFHNDARXLF3LIV2GVMKUNV2MWC6USUV2Q, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.476491837 (540h15m12.347574872s) [noise=true , address=45.91.116.7:4010 , piece_id=PCGK7VML6G23UPAGRB73ARAKWKLIN2OMYAKHHGA4QUWO24JATGMA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.516874483 (540h15m12.387957518s) [noise=true , address=77.90.31.14:4663 , piece_id=4G7NBQFYRVSFPB7YWU3HQWKARO5OFGIR5HQZ4T3Y5J36WFKLEXOA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.554794398 (540h15m12.425877464s) [noise=true , address=77.90.31.13:4629 , piece_id=3JPQRVWSK6UAODOAUU5VYGUF33ZN6KCQEFCZLZVDTZJ7IOLMW3XQ, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.568603884 (540h15m12.439686919s) [noise=true , address=45.91.116.13:4153 , piece_id=W6QNHQJV4RBIQFBGJEJX7GP67NCSE2DRGSXPVK6PA36MJCWONEHA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.583888138 (540h15m12.454971173s) [noise=true , address=45.91.116.6:4973 , piece_id=DBXF5T7HZGGIOQI64SPLSLOOQNN6PJD45BBQX5FEG3ITTJXWSVEA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: <nil> Dec 28 03:44:12.583906423 (540h15m12.454989469s) [seg_pos={0 0}, upload=473797]: Segment reached optimal threshold of 13 pieces. Dec 28 03:44:12.584082679 (540h15m12.455165714s) [noise=true , address=145.79.227.39:4014 , piece_id=WLR7UYS5KL2PX7OPDLX7L7MFMCK6EF2JOWETAYUVP2D66NN33HNA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: expected piece hash; context canceled; EOF Dec 28 03:44:12.584088520 (540h15m12.455171555s) [noise=true , address=145.79.227.39:4014 , piece_id=WLR7UYS5KL2PX7OPDLX7L7MFMCK6EF2JOWETAYUVP2D66NN33HNA, seg_pos={0 0}, upload=473797]: Long tail cancellation due to optimal threshold Dec 28 03:44:12.584197337 (540h15m12.455280372s) [noise=true , address=45.91.116.8:4048 , piece_id=3RB3HZLKQXH47OJ4ZYRCGNYFVIFNZVNPIPVR4O5QNW2MUJZ33IKA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 28 03:44:12.584202427 (540h15m12.455285472s) [noise=true , address=45.91.116.8:4048 , piece_id=3RB3HZLKQXH47OJ4ZYRCGNYFVIFNZVNPIPVR4O5QNW2MUJZ33IKA, seg_pos={0 0}, upload=473797]: Long tail cancellation due to optimal threshold Dec 28 03:44:12.584452663 (540h15m12.455535708s) [noise=true , address=77.90.31.12:4590 , piece_id=STSUMBGCMWREK7MHCR3TYPYGT2QPWBJCMPHDGFBRSAECUJ2FFWRA, seg_pos={0 0}, upload=473797]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 03:44:12.584472551 (540h15m12.455555586s) [noise=true , address=77.90.31.12:4590 , piece_id=STSUMBGCMWREK7MHCR3TYPYGT2QPWBJCMPHDGFBRSAECUJ2FFWRA, seg_pos={0 0}, upload=473797]: Long tail cancellation due to optimal threshold Dec 28 03:44:12.584509241 (540h15m12.455592286s) [seg_pos={0 0}, upload=473797]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:44:12.584616625 (540h15m12.455699671s) [upload=473797]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:44:12.584624210 (540h15m12.455707255s) [upload=473797]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:44:12.894137406 (540h15m12.765220441s) [upload=473797]: Done uploading segments... Dec 28 03:44:12.894160861 (540h15m12.765243906s) [upload=473797]: Upload finished. err: <nil> Duration: 30.002502349s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.289885763s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:44:29 UTC) orbiter warning: context deadline exceeded Duration: 6.289909927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 45.248244861s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:44:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 45.248276723s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...X...........X.....XX.....X..X.X....X.XXXX....X.X.X...X..X] duration: 5.00145699s trace: https://jaeger.dev.storj.io/trace/202aa1036c478d60 FAIL (2025-12-28 03:44:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001473111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.68833714s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:43:55 UTC) orbiter warning: context deadline exceeded Duration: 6.688495261s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 10.191707881s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:43:35 UTC) orbiter warning: context deadline exceeded Duration: 10.191783054s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [X.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXXX] duration: 30.00285385s trace: https://jaeger.dev.storj.io/trace/885518907fd220 FAIL (2025-12-28 03:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: QRYXO6YAT3ZIDM3OFCKHLK4X3A7IKSHVBPNKHTFLXQE4XSGB5ASA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: TSXGUCOWDZFQ5PYZL32PYJFZ6XXRJVADCLZMCE2YEAKQMQ3IACFA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: I7RJZWGQ7QJGPZUFDF2DUUPRHDPBHKGQYOHYPR56LHLTESQ4XEPA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 3SKZNZFFRNDOTS2I6KNLM7RJ46UPRRUU2TGN6Y5BYWYCBPUIE4FA) context deadline exceeded ======LOGS====== Dec 28 03:42:57.214081396 (540h13m57.085164441s) [upload=473777]: Starting upload Dec 28 03:42:57.214162721 (540h13m57.085245776s) [upload=473777]: Done starting upload Dec 28 03:42:57.214421714 (540h13m57.085504759s) [upload=473777]: Uploading segments... Dec 28 03:42:57.214681329 (540h13m57.085764364s) [upload=473777]: Got next segment. Inline: false Dec 28 03:42:57.214742405 (540h13m57.08582545s) [upload=473777]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:42:57.214749308 (540h13m57.085832353s) [upload=473777]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:42:57.514431223 (540h13m57.385514278s) [seg_pos={0 0}, upload=473777]: Begin upload segment... Dec 28 03:42:57.514501647 (540h13m57.385584682s) [seg_pos={0 0}, upload=473777]: Done begin upload segment. Dec 28 03:42:57.514531144 (540h13m57.385614189s) [noise=true , address=145.79.227.46:4602 , piece_id=YGFCUN5KL7ITDOOVMNVVFM43XSLAIYU574KZTYAPHYEBHCUP5PNA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.514537085 (540h13m57.38562013s) [noise=true , address=145.79.227.3:4150 , piece_id=HVIWXG55JYIKCIFVJBJM2KTJ73ME55EJ23LYR6EYWZYZET4WRY5Q, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.515399978 (540h13m57.386483023s) [noise=true , address=45.91.116.8:4045 , piece_id=3SKZNZFFRNDOTS2I6KNLM7RJ46UPRRUU2TGN6Y5BYWYCBPUIE4FA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.515868851 (540h13m57.386951886s) [noise=true , address=145.79.227.15:4874 , piece_id=IVQ2GE5VAIYVCCRMN2MSDCLTRNW6II3PE4TFZIV3H6EON3BHGFDQ, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.515930939 (540h13m57.387013994s) [noise=true , address=45.91.116.4:4939 , piece_id=JTZXR3X747T2MZOH4VPIX42DORVU3BIP56MJZCISCSNDHHW3IINA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.516278280 (540h13m57.387361325s) [noise=true , address=145.79.227.50:4373 , piece_id=MNNYSZB6YOTTW5OLBAVGZJCXJZB7ZZZH45WSVO4B62SL6GVJTEPA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.516774686 (540h13m57.387857731s) [noise=true , address=45.91.116.6:4976 , piece_id=QRYXO6YAT3ZIDM3OFCKHLK4X3A7IKSHVBPNKHTFLXQE4XSGB5ASA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.516893472 (540h13m57.387976517s) [noise=true , address=145.79.227.11:4732 , piece_id=CNKL4PJG6FEAHE3I5PEIO6TZEBTCEBNZ5RM25P5BKB5POJMIDLQQ, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.517294246 (540h13m57.388377291s) [noise=true , address=45.91.116.7:4009 , piece_id=N5CX4VBDRNIPOXVZNCN7POB5PXBUID2KEDMBYIKNGQWDSOCZEFZQ, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.517729133 (540h13m57.388812178s) [noise=true , address=45.91.116.14:4189 , piece_id=KBVKN5BW46XV6D7BUZJ3EM57IMCQVUYB3LAQBFAROZ6BFM4US2YA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.518683291 (540h13m57.389766336s) [noise=true , address=45.91.116.9:4264 , piece_id=MI4OZM3P2LUOWFGBVH7WQS6MX3MB6IEGM5PE6Q6HKVPH57NSYKJA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.519067242 (540h13m57.390150287s) [noise=true , address=145.79.227.39:4014 , piece_id=S7NXLT5GQ5QBVNP7TZ2NFQU5THESBMSEAJNQ3O6WRCSSGM6VBKGQ, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.519124240 (540h13m57.390207275s) [noise=true , address=77.90.31.15:4700 , piece_id=TSXGUCOWDZFQ5PYZL32PYJFZ6XXRJVADCLZMCE2YEAKQMQ3IACFA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.519652125 (540h13m57.39073516s) [noise=true , address=145.79.227.31:4798 , piece_id=664TYYF4ILWFYAL2KTBS2AHOD6HZVV7RJMSHR633GAXK2G44PHDA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.520149493 (540h13m57.391232528s) [noise=true , address=45.91.116.10:4082 , piece_id=SIPPYXVRWKXTRMDXFGDY5C5W6WXQWAGWEHXYXAUUGNPUE3TUPGHA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:57.520543382 (540h13m57.391626427s) [noise=true , address=77.90.31.19:4841 , piece_id=I7RJZWGQ7QJGPZUFDF2DUUPRHDPBHKGQYOHYPR56LHLTESQ4XEPA, seg_pos={0 0}, upload=473777]: Uploading piece... Dec 28 03:42:59.172634597 (540h13m59.043717642s) [upload=473777]: Next returned nil segment Dec 28 03:42:59.172650628 (540h13m59.043733673s) [upload=473777]: Waiting for error group managing segments... Dec 28 03:42:59.518841334 (540h13m59.389924369s) [noise=true , address=145.79.227.46:4602 , piece_id=YGFCUN5KL7ITDOOVMNVVFM43XSLAIYU574KZTYAPHYEBHCUP5PNA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.519580972 (540h13m59.390664017s) [noise=true , address=145.79.227.39:4014 , piece_id=S7NXLT5GQ5QBVNP7TZ2NFQU5THESBMSEAJNQ3O6WRCSSGM6VBKGQ, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.531855686 (540h13m59.402938721s) [noise=true , address=145.79.227.15:4874 , piece_id=IVQ2GE5VAIYVCCRMN2MSDCLTRNW6II3PE4TFZIV3H6EON3BHGFDQ, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.540449049 (540h13m59.411532084s) [noise=true , address=145.79.227.50:4373 , piece_id=MNNYSZB6YOTTW5OLBAVGZJCXJZB7ZZZH45WSVO4B62SL6GVJTEPA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.541488579 (540h13m59.412571624s) [noise=true , address=145.79.227.31:4798 , piece_id=664TYYF4ILWFYAL2KTBS2AHOD6HZVV7RJMSHR633GAXK2G44PHDA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.701140052 (540h13m59.572223087s) [noise=true , address=77.90.31.19:4841 , piece_id=I7RJZWGQ7QJGPZUFDF2DUUPRHDPBHKGQYOHYPR56LHLTESQ4XEPA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.718725909 (540h13m59.589808954s) [noise=true , address=77.90.31.15:4700 , piece_id=TSXGUCOWDZFQ5PYZL32PYJFZ6XXRJVADCLZMCE2YEAKQMQ3IACFA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.757575655 (540h13m59.6286587s) [noise=true , address=45.91.116.10:4082 , piece_id=SIPPYXVRWKXTRMDXFGDY5C5W6WXQWAGWEHXYXAUUGNPUE3TUPGHA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.759221379 (540h13m59.630304414s) [noise=true , address=45.91.116.8:4045 , piece_id=3SKZNZFFRNDOTS2I6KNLM7RJ46UPRRUU2TGN6Y5BYWYCBPUIE4FA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.774287798 (540h13m59.645370853s) [noise=true , address=45.91.116.7:4009 , piece_id=N5CX4VBDRNIPOXVZNCN7POB5PXBUID2KEDMBYIKNGQWDSOCZEFZQ, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.793921694 (540h13m59.665004739s) [noise=true , address=45.91.116.6:4976 , piece_id=QRYXO6YAT3ZIDM3OFCKHLK4X3A7IKSHVBPNKHTFLXQE4XSGB5ASA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.804881153 (540h13m59.675964198s) [noise=true , address=145.79.227.11:4732 , piece_id=CNKL4PJG6FEAHE3I5PEIO6TZEBTCEBNZ5RM25P5BKB5POJMIDLQQ, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.890528991 (540h13m59.761612026s) [noise=true , address=45.91.116.14:4189 , piece_id=KBVKN5BW46XV6D7BUZJ3EM57IMCQVUYB3LAQBFAROZ6BFM4US2YA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: <nil> Dec 28 03:42:59.890546615 (540h13m59.76162966s) [seg_pos={0 0}, upload=473777]: Segment reached optimal threshold of 13 pieces. Dec 28 03:42:59.890795138 (540h13m59.761878173s) [noise=true , address=45.91.116.9:4264 , piece_id=MI4OZM3P2LUOWFGBVH7WQS6MX3MB6IEGM5PE6Q6HKVPH57NSYKJA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 28 03:42:59.890802041 (540h13m59.761885086s) [noise=true , address=45.91.116.9:4264 , piece_id=MI4OZM3P2LUOWFGBVH7WQS6MX3MB6IEGM5PE6Q6HKVPH57NSYKJA, seg_pos={0 0}, upload=473777]: Long tail cancellation due to optimal threshold Dec 28 03:42:59.891007793 (540h13m59.762090838s) [noise=true , address=145.79.227.3:4150 , piece_id=HVIWXG55JYIKCIFVJBJM2KTJ73ME55EJ23LYR6EYWZYZET4WRY5Q, seg_pos={0 0}, upload=473777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 28 03:42:59.891013894 (540h13m59.762096929s) [noise=true , address=145.79.227.3:4150 , piece_id=HVIWXG55JYIKCIFVJBJM2KTJ73ME55EJ23LYR6EYWZYZET4WRY5Q, seg_pos={0 0}, upload=473777]: Long tail cancellation due to optimal threshold Dec 28 03:42:59.891210880 (540h13m59.762293915s) [noise=true , address=45.91.116.4:4939 , piece_id=JTZXR3X747T2MZOH4VPIX42DORVU3BIP56MJZCISCSNDHHW3IINA, seg_pos={0 0}, upload=473777]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 03:42:59.891230196 (540h13m59.762313241s) [noise=true , address=45.91.116.4:4939 , piece_id=JTZXR3X747T2MZOH4VPIX42DORVU3BIP56MJZCISCSNDHHW3IINA, seg_pos={0 0}, upload=473777]: Long tail cancellation due to optimal threshold Dec 28 03:42:59.891254212 (540h13m59.762337257s) [seg_pos={0 0}, upload=473777]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:42:59.891364632 (540h13m59.762447677s) [upload=473777]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:42:59.891372116 (540h13m59.762455161s) [upload=473777]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:43:00.236161063 (540h14m0.107244098s) [upload=473777]: Done uploading segments... Dec 28 03:43:00.236182625 (540h14m0.10726566s) [upload=473777]: Upload finished. err: <nil> Duration: 30.002878356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 46.066683134s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:43:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 46.066698493s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.379156507s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:43:14 UTC) orbiter warning: context deadline exceeded Duration: 6.379185833s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [....X...........X.....XX.....X..X.X....X.XXXX....X.X.X...X..] duration: 4.628488084s trace: https://jaeger.dev.storj.io/trace/4c869103fb81995b FAIL (2025-12-28 03:43:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.628512141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXXX] duration: 30.002739728s trace: https://jaeger.dev.storj.io/trace/742e94f81a1edba FAIL (2025-12-28 03:42:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: HZHR5IZRTCRYZYTOUJ5Y7AESDWN65UQFCN3DYW6LD3BGEPOEQHQA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: CVKKRMDNF7S6IAGHN6EGC24IFI45JCSOVB3NXAQ5YMLUWYQHRCYQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: HYY73EYGJ5HF76RJYNCULFJD4FEKRMAOTEEFFFIYRQA3VKO3OTHQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: Q2YOYVMWAAQKOGRVF6YPYB253JPL2X4YRRUZC4JEKCQU3RNV3OWA) context deadline exceeded ======LOGS====== 42:18.950639686 (540h13m18.821722731s) [upload=473768]: Starting upload Dec 28 03:42:18.950732993 (540h13m18.821816028s) [upload=473768]: Done starting upload Dec 28 03:42:18.950862199 (540h13m18.821945244s) [upload=473768]: Uploading segments... Dec 28 03:42:18.950986356 (540h13m18.822069391s) [upload=473768]: Got next segment. Inline: false Dec 28 03:42:18.951035349 (540h13m18.822118394s) [upload=473768]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:42:18.951042252 (540h13m18.822125307s) [upload=473768]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:42:19.209025966 (540h13m19.080109051s) [seg_pos={0 0}, upload=473768]: Begin upload segment... Dec 28 03:42:19.209087443 (540h13m19.080170479s) [seg_pos={0 0}, upload=473768]: Done begin upload segment. Dec 28 03:42:19.209108513 (540h13m19.080191558s) [noise=true , address=45.91.116.5:4300 , piece_id=Q2YOYVMWAAQKOGRVF6YPYB253JPL2X4YRRUZC4JEKCQU3RNV3OWA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.209126157 (540h13m19.080209192s) [noise=true , address=77.90.31.14:4661 , piece_id=FDC76LXGL7TU7YRYZ6BF255ANOLXVOECPJJFITEZQWTVXHWBTDGQ, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.209635127 (540h13m19.080718172s) [noise=true , address=145.79.227.37:4565 , piece_id=NEFF25LXNVIIA7EB2UJKNN4K5Z5WCUBLREYFLPGXXADUQDOEDQDA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.210211644 (540h13m19.081294729s) [noise=true , address=145.79.227.32:4835 , piece_id=4NFONV7F4ZVW7ODIC26A6IRXUXKSBHB24YDTJWGVYQSMYK27425A, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.210281116 (540h13m19.081364151s) [noise=true , address=45.91.116.13:4156 , piece_id=HZHR5IZRTCRYZYTOUJ5Y7AESDWN65UQFCN3DYW6LD3BGEPOEQHQA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.210744789 (540h13m19.081827824s) [noise=true , address=145.79.227.31:4799 , piece_id=3J3HRL7I47ZLLR7ZZL57KG5OOPKEYMAF5BVL6WLPRJBUTKTJ5IGQ, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.210763855 (540h13m19.081846891s) [noise=true , address=145.79.227.36:4518 , piece_id=EOIDVC45FX75MZX2RSW6EDRR26GXQDA2USIP5GTWZ4IDVOJBXZPA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.211165942 (540h13m19.082248987s) [noise=true , address=77.90.31.17:4770 , piece_id=CVKKRMDNF7S6IAGHN6EGC24IFI45JCSOVB3NXAQ5YMLUWYQHRCYQ, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.211180399 (540h13m19.082263434s) [noise=true , address=77.90.31.19:4842 , piece_id=W3SEIHQFYGWN67VPA3H6DXRVMDL5OL7KTTQLFNP5NT7J7NQCEOWA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.211506078 (540h13m19.082589114s) [noise=true , address=145.79.227.40:4051 , piece_id=6LDRFMO5WSM2UQBWUPXCU36CERAITDXM33O3FOWADWZEKXACUS5A, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.211839404 (540h13m19.082922449s) [noise=true , address=145.79.227.11:4729 , piece_id=W65TM6B2OYC2P5CLKZWCVLEFXEFN4CZTXQY7ZRDWY6VK474DWTZA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.212148542 (540h13m19.083231577s) [noise=true , address=45.91.116.7:4012 , piece_id=6XRFI2V5ERALWSC35SUCDWAL5C4TDGA7WVSMSX2SQ7UIKZ6X3DGA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.212472048 (540h13m19.083555093s) [noise=true , address=145.79.227.44:4195 , piece_id=KLFOH3MFSCLSZHX5SL6EL7H3Y2U2F3ZN62ZM3BQFCSDOKUACA7LA, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.212724458 (540h13m19.083807494s) [noise=true , address=45.91.116.14:4192 , piece_id=TTHVL3YIQYBXSPNFAOWTMBWVMMZYVHSNFKF6AK2FZBP4JF5A3JOQ, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.212806245 (540h13m19.08388928s) [noise=true , address=145.79.227.7:4293 , piece_id=75ARWKJYP4CMC6X3KEKP5BPWGLTHOFMFJJDSX3FEBES44TT3773A, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:19.213118069 (540h13m19.084201104s) [noise=true , address=45.91.116.6:4973 , piece_id=HYY73EYGJ5HF76RJYNCULFJD4FEKRMAOTEEFFFIYRQA3VKO3OTHQ, seg_pos={0 0}, upload=473768]: Uploading piece... Dec 28 03:42:20.947918626 (540h13m20.819001671s) [upload=473768]: Next returned nil segment Dec 28 03:42:20.947949245 (540h13m20.81903228s) [upload=473768]: Waiting for error group managing segments... Dec 28 03:42:21.237580382 (540h13m21.108663417s) [noise=true , address=145.79.227.37:4565 , piece_id=NEFF25LXNVIIA7EB2UJKNN4K5Z5WCUBLREYFLPGXXADUQDOEDQDA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.325619293 (540h13m21.196702338s) [noise=true , address=145.79.227.32:4835 , piece_id=4NFONV7F4ZVW7ODIC26A6IRXUXKSBHB24YDTJWGVYQSMYK27425A, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.326989714 (540h13m21.198072749s) [noise=true , address=145.79.227.36:4518 , piece_id=EOIDVC45FX75MZX2RSW6EDRR26GXQDA2USIP5GTWZ4IDVOJBXZPA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.336478411 (540h13m21.207561476s) [noise=true , address=145.79.227.11:4729 , piece_id=W65TM6B2OYC2P5CLKZWCVLEFXEFN4CZTXQY7ZRDWY6VK474DWTZA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.412289348 (540h13m21.283372393s) [noise=true , address=77.90.31.17:4770 , piece_id=CVKKRMDNF7S6IAGHN6EGC24IFI45JCSOVB3NXAQ5YMLUWYQHRCYQ, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.441371576 (540h13m21.312454611s) [noise=true , address=45.91.116.13:4156 , piece_id=HZHR5IZRTCRYZYTOUJ5Y7AESDWN65UQFCN3DYW6LD3BGEPOEQHQA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.460787818 (540h13m21.331870853s) [noise=true , address=45.91.116.6:4973 , piece_id=HYY73EYGJ5HF76RJYNCULFJD4FEKRMAOTEEFFFIYRQA3VKO3OTHQ, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.467015546 (540h13m21.338098581s) [noise=true , address=45.91.116.7:4012 , piece_id=6XRFI2V5ERALWSC35SUCDWAL5C4TDGA7WVSMSX2SQ7UIKZ6X3DGA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.516538848 (540h13m21.387621893s) [noise=true , address=77.90.31.19:4842 , piece_id=W3SEIHQFYGWN67VPA3H6DXRVMDL5OL7KTTQLFNP5NT7J7NQCEOWA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.522825638 (540h13m21.393908683s) [noise=true , address=77.90.31.14:4661 , piece_id=FDC76LXGL7TU7YRYZ6BF255ANOLXVOECPJJFITEZQWTVXHWBTDGQ, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.566296504 (540h13m21.437379539s) [noise=true , address=45.91.116.14:4192 , piece_id=TTHVL3YIQYBXSPNFAOWTMBWVMMZYVHSNFKF6AK2FZBP4JF5A3JOQ, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:21.954336659 (540h13m21.825419694s) [noise=true , address=45.91.116.5:4300 , piece_id=Q2YOYVMWAAQKOGRVF6YPYB253JPL2X4YRRUZC4JEKCQU3RNV3OWA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:22.433560593 (540h13m22.304643638s) [noise=true , address=145.79.227.31:4799 , piece_id=3J3HRL7I47ZLLR7ZZL57KG5OOPKEYMAF5BVL6WLPRJBUTKTJ5IGQ, seg_pos={0 0}, upload=473768]: Done uploading piece. err: <nil> Dec 28 03:42:22.433584710 (540h13m22.304667755s) [seg_pos={0 0}, upload=473768]: Segment reached optimal threshold of 13 pieces. Dec 28 03:42:22.433844233 (540h13m22.304927278s) [noise=true , address=145.79.227.7:4293 , piece_id=75ARWKJYP4CMC6X3KEKP5BPWGLTHOFMFJJDSX3FEBES44TT3773A, seg_pos={0 0}, upload=473768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:42156->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:42156->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42156->145.79.227.7:4293: use of closed network connection Dec 28 03:42:22.433854123 (540h13m22.304937158s) [noise=true , address=145.79.227.7:4293 , piece_id=75ARWKJYP4CMC6X3KEKP5BPWGLTHOFMFJJDSX3FEBES44TT3773A, seg_pos={0 0}, upload=473768]: Long tail cancellation due to optimal threshold Dec 28 03:42:22.433998498 (540h13m22.305081533s) [noise=true , address=145.79.227.40:4051 , piece_id=6LDRFMO5WSM2UQBWUPXCU36CERAITDXM33O3FOWADWZEKXACUS5A, seg_pos={0 0}, upload=473768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 03:42:22.434014287 (540h13m22.305097322s) [noise=true , address=145.79.227.40:4051 , piece_id=6LDRFMO5WSM2UQBWUPXCU36CERAITDXM33O3FOWADWZEKXACUS5A, seg_pos={0 0}, upload=473768]: Long tail cancellation due to optimal threshold Dec 28 03:42:22.434156999 (540h13m22.305240034s) [noise=true , address=145.79.227.44:4195 , piece_id=KLFOH3MFSCLSZHX5SL6EL7H3Y2U2F3ZN62ZM3BQFCSDOKUACA7LA, seg_pos={0 0}, upload=473768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 03:42:22.434162589 (540h13m22.305245624s) [noise=true , address=145.79.227.44:4195 , piece_id=KLFOH3MFSCLSZHX5SL6EL7H3Y2U2F3ZN62ZM3BQFCSDOKUACA7LA, seg_pos={0 0}, upload=473768]: Long tail cancellation due to optimal threshold Dec 28 03:42:22.434187828 (540h13m22.305270863s) [seg_pos={0 0}, upload=473768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:42:22.434276497 (540h13m22.305359542s) [upload=473768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:42:22.434282278 (540h13m22.305365313s) [upload=473768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:42:22.729689032 (540h13m22.600772067s) [upload=473768]: Done uploading segments... Dec 28 03:42:22.729708870 (540h13m22.600791905s) [upload=473768]: Upload finished. err: <nil> Duration: 30.002760978s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.173206382s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:42:48 UTC) orbiter warning: context deadline exceeded Duration: 6.173230528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...X......................] duration: 30.415284545s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:42:33 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 30.415314803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration: 39.169425438s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:41:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 39.169446149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X....X...........X.....XX.....X..X.X....X.XXXX....X.X.X...X.] duration: 3.380126928s trace: https://jaeger.dev.storj.io/trace/4c869103fb81995b FAIL (2025-12-28 03:41:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.380162716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 12.97456298s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:41:50 UTC) orbiter warning: context deadline exceeded Duration: 12.97461055s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX] duration: 30.003692137s trace: https://jaeger.dev.storj.io/trace/485f0f3573a471b5 FAIL (2025-12-28 03:41:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 77YIVT6DWDL3UNBYHKMEI5ECXK74C3ZRHM6FMAESTWTF27IGLKSA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 2U4KQDLU4WUYUJRT4RUHKM2CZHQSEXBX454YAEUHVSWTB4KZMVFA) context deadline exceeded ======LOGS====== 4.299490239s) [upload=473747]: Uploading segments... Dec 28 03:41:04.428612896 (540h12m4.299695941s) [upload=473747]: Got next segment. Inline: false Dec 28 03:41:04.428672649 (540h12m4.299755694s) [upload=473747]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:41:04.428679302 (540h12m4.299762347s) [upload=473747]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:41:04.675531660 (540h12m4.546614705s) [seg_pos={0 0}, upload=473747]: Begin upload segment... Dec 28 03:41:04.675595061 (540h12m4.546678106s) [seg_pos={0 0}, upload=473747]: Done begin upload segment. Dec 28 03:41:04.675711352 (540h12m4.546794387s) [noise=true , address=145.79.227.15:4873 , piece_id=UHP2ARFBACSWJR76ELLCYMQTZWQXTAFQ7T4G4PFQM2ZJT4AOK3IA, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.675732743 (540h12m4.546815778s) [noise=true , address=45.91.116.13:4156 , piece_id=77YIVT6DWDL3UNBYHKMEI5ECXK74C3ZRHM6FMAESTWTF27IGLKSA, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.676434930 (540h12m4.547517975s) [noise=true , address=77.90.31.14:4663 , piece_id=AAOBNJOWD3LX5JG43LVCLZOU7JTNBCDOB4KUUWNVGKCZYHPEULOA, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.676824762 (540h12m4.547907797s) [noise=true , address=145.79.227.16:4911 , piece_id=PBWPYI6UQRW2NLFRIDQ4XHT4NCPFEWXEDUEA3KQ5WGQH7FGXGLNQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.677239642 (540h12m4.548322687s) [noise=true , address=145.79.227.37:4567 , piece_id=2AYO3NCFP6TG22AEIPXTZJEOKRXKQYMRTY5OMHN4ZAW6MXVK6TXQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.677581152 (540h12m4.548664197s) [noise=true , address=145.79.227.27:4691 , piece_id=Z2GLDMKA6EVWIGYXO7ZC7SJJTLZJKJW2HPNPEHCNJRQUQTHFK4HQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.686593143 (540h12m4.557676198s) [noise=true , address=145.79.227.42:4121 , piece_id=JFXTFYX43BGFD6AQCDTSBA5I5DSLNKFUESNFY7F4LDMDWFVKOIRQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.687587968 (540h12m4.558671013s) [noise=true , address=145.79.227.39:4014 , piece_id=XUFSYPV5IZMTTAAPINT22TCRW6V63O5OIV2PP6WK5V5TEWEDOTZQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.695244507 (540h12m4.566327552s) [noise=true , address=145.79.227.33:4871 , piece_id=JLNCTHOD5K54MFUVXONTDE44HROT3XAESZJ5WDF4PUEQIEEAVSBQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.698845423 (540h12m4.569928458s) [noise=true , address=45.91.116.9:4261 , piece_id=2U4KQDLU4WUYUJRT4RUHKM2CZHQSEXBX454YAEUHVSWTB4KZMVFA, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.700991951 (540h12m4.572075006s) [noise=true , address=45.91.116.4:4940 , piece_id=S3EYFPSWPZWHP2HA5CEXDIW7GZRYN664GLS5YNNI4EL73GPQNSHQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.702381717 (540h12m4.573464753s) [noise=true , address=145.79.227.51:4410 , piece_id=LS3ZQPQ4EE7QAEWI667TERDD67FHJ4MKSXNJKJ5LHHCDF3X67UPQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.705293802 (540h12m4.576376847s) [noise=true , address=45.91.116.5:4298 , piece_id=KQM2MFX2PRRLCNXSXYSV3NH3TWV2OJQSE3FUXJLA5VWNF7WKQ6IQ, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.709784002 (540h12m4.580867047s) [noise=true , address=45.91.116.7:4010 , piece_id=236SP6BSYUHULKVY2XIH6EMHZC4WGD3MUMOFDZQN5GAMIL7Q5R6Q, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.710413050 (540h12m4.581496095s) [noise=true , address=145.79.227.11:4729 , piece_id=XZOKHCXNR3NOIK5FJAOTF6WISRD2CW3ILIZMUJOFBO34OJMIFI7Q, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:04.712911578 (540h12m4.583994623s) [noise=true , address=45.91.116.8:4045 , piece_id=CQIRTUDXXK5UMW3U5P7Z44BFO25VUZ35KE2COJZWI2FNJOHBK3VA, seg_pos={0 0}, upload=473747]: Uploading piece... Dec 28 03:41:06.429578979 (540h12m6.300662025s) [upload=473747]: Next returned nil segment Dec 28 03:41:06.429595822 (540h12m6.300678857s) [upload=473747]: Waiting for error group managing segments... Dec 28 03:41:06.841287141 (540h12m6.712370176s) [noise=true , address=145.79.227.42:4121 , piece_id=JFXTFYX43BGFD6AQCDTSBA5I5DSLNKFUESNFY7F4LDMDWFVKOIRQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.868312922 (540h12m6.739395957s) [noise=true , address=145.79.227.33:4871 , piece_id=JLNCTHOD5K54MFUVXONTDE44HROT3XAESZJ5WDF4PUEQIEEAVSBQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.868822984 (540h12m6.739906039s) [noise=true , address=145.79.227.15:4873 , piece_id=UHP2ARFBACSWJR76ELLCYMQTZWQXTAFQ7T4G4PFQM2ZJT4AOK3IA, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.873340436 (540h12m6.744423481s) [noise=true , address=145.79.227.27:4691 , piece_id=Z2GLDMKA6EVWIGYXO7ZC7SJJTLZJKJW2HPNPEHCNJRQUQTHFK4HQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.877238639 (540h12m6.748321674s) [noise=true , address=145.79.227.11:4729 , piece_id=XZOKHCXNR3NOIK5FJAOTF6WISRD2CW3ILIZMUJOFBO34OJMIFI7Q, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.877870362 (540h12m6.748953397s) [noise=true , address=145.79.227.37:4567 , piece_id=2AYO3NCFP6TG22AEIPXTZJEOKRXKQYMRTY5OMHN4ZAW6MXVK6TXQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.879515734 (540h12m6.75059877s) [noise=true , address=145.79.227.39:4014 , piece_id=XUFSYPV5IZMTTAAPINT22TCRW6V63O5OIV2PP6WK5V5TEWEDOTZQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.898339108 (540h12m6.769422153s) [noise=true , address=45.91.116.4:4940 , piece_id=S3EYFPSWPZWHP2HA5CEXDIW7GZRYN664GLS5YNNI4EL73GPQNSHQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.930221436 (540h12m6.801304481s) [noise=true , address=77.90.31.14:4663 , piece_id=AAOBNJOWD3LX5JG43LVCLZOU7JTNBCDOB4KUUWNVGKCZYHPEULOA, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.935904297 (540h12m6.806987332s) [noise=true , address=45.91.116.7:4010 , piece_id=236SP6BSYUHULKVY2XIH6EMHZC4WGD3MUMOFDZQN5GAMIL7Q5R6Q, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:06.973947117 (540h12m6.845030162s) [noise=true , address=45.91.116.13:4156 , piece_id=77YIVT6DWDL3UNBYHKMEI5ECXK74C3ZRHM6FMAESTWTF27IGLKSA, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:07.031865324 (540h12m6.902948359s) [noise=true , address=45.91.116.9:4261 , piece_id=2U4KQDLU4WUYUJRT4RUHKM2CZHQSEXBX454YAEUHVSWTB4KZMVFA, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:07.232333689 (540h12m7.103416725s) [noise=true , address=145.79.227.51:4410 , piece_id=LS3ZQPQ4EE7QAEWI667TERDD67FHJ4MKSXNJKJ5LHHCDF3X67UPQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: <nil> Dec 28 03:41:07.232354218 (540h12m7.103437253s) [seg_pos={0 0}, upload=473747]: Segment reached optimal threshold of 13 pieces. Dec 28 03:41:07.232576211 (540h12m7.103659246s) [noise=true , address=45.91.116.8:4045 , piece_id=CQIRTUDXXK5UMW3U5P7Z44BFO25VUZ35KE2COJZWI2FNJOHBK3VA, seg_pos={0 0}, upload=473747]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:44716->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:44716->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44716->45.91.116.8:4045: use of closed network connection Dec 28 03:41:07.232584156 (540h12m7.103667201s) [noise=true , address=45.91.116.8:4045 , piece_id=CQIRTUDXXK5UMW3U5P7Z44BFO25VUZ35KE2COJZWI2FNJOHBK3VA, seg_pos={0 0}, upload=473747]: Long tail cancellation due to optimal threshold Dec 28 03:41:07.232692502 (540h12m7.103775537s) [noise=true , address=45.91.116.5:4298 , piece_id=KQM2MFX2PRRLCNXSXYSV3NH3TWV2OJQSE3FUXJLA5VWNF7WKQ6IQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 03:41:07.232714535 (540h12m7.10379759s) [noise=true , address=45.91.116.5:4298 , piece_id=KQM2MFX2PRRLCNXSXYSV3NH3TWV2OJQSE3FUXJLA5VWNF7WKQ6IQ, seg_pos={0 0}, upload=473747]: Long tail cancellation due to optimal threshold Dec 28 03:41:07.232850513 (540h12m7.103933558s) [noise=true , address=145.79.227.16:4911 , piece_id=PBWPYI6UQRW2NLFRIDQ4XHT4NCPFEWXEDUEA3KQ5WGQH7FGXGLNQ, seg_pos={0 0}, upload=473747]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: write tcp 188.166.225.170:54064->145.79.227.16:4911: use of closed network connection; write tcp 188.166.225.170:54064->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54064->145.79.227.16:4911: use of closed network connection Dec 28 03:41:07.232858789 (540h12m7.103941824s) [noise=true , address=145.79.227.16:4911 , piece_id=PBWPYI6UQRW2NLFRIDQ4XHT4NCPFEWXEDUEA3KQ5WGQH7FGXGLNQ, seg_pos={0 0}, upload=473747]: Long tail cancellation due to optimal threshold Dec 28 03:41:07.232876342 (540h12m7.103959377s) [seg_pos={0 0}, upload=473747]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:41:07.232941356 (540h12m7.104024391s) [upload=473747]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:41:07.232945293 (540h12m7.104028329s) [upload=473747]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:41:07.531981459 (540h12m7.403064504s) [upload=473747]: Done uploading segments... Dec 28 03:41:07.532004382 (540h12m7.403087427s) [upload=473747]: Upload finished. err: <nil> Duration: 30.00370917s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001124908s trace: https://jaeger.dev.storj.io/trace/3b8130f32e60318d FAIL (2025-12-28 03:41:27 UTC) orbiter warning: context deadline exceeded Duration: 30.001137833s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 53.540549028s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:41:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 53.540571129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.X....X...........X.....XX.....X..X.X....X.XXXX....X.X.X...X] duration: 5.001677169s trace: https://jaeger.dev.storj.io/trace/4c869103fb81995b FAIL (2025-12-28 03:40:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001701597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXXX] duration: 30.003975839s trace: https://jaeger.dev.storj.io/trace/76323db66a054dc8 FAIL (2025-12-28 03:40:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:40:04.583674735 (540h11m4.45475778s) [upload=473731]: Starting upload Dec 28 03:40:04.583772411 (540h11m4.454855446s) [upload=473731]: Done starting upload Dec 28 03:40:04.583872501 (540h11m4.454955546s) [upload=473731]: Uploading segments... Dec 28 03:40:04.584113149 (540h11m4.455196195s) [upload=473731]: Got next segment. Inline: false Dec 28 03:40:04.584169958 (540h11m4.455253003s) [upload=473731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:40:04.584176610 (540h11m4.455259676s) [upload=473731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:40:04.829302962 (540h11m4.700386007s) [seg_pos={0 0}, upload=473731]: Begin upload segment... Dec 28 03:40:04.829364950 (540h11m4.700448626s) [seg_pos={0 0}, upload=473731]: Done begin upload segment. Dec 28 03:40:04.829391801 (540h11m4.700474846s) [noise=true , address=145.79.227.39:4014 , piece_id=RHDZ4USCKOYCC27JSN23TXXPKFLDVLQEL4USCDHBHKFYNLDFZPEA, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.829411749 (540h11m4.700494794s) [noise=true , address=145.79.227.42:4123 , piece_id=EY6AQNGX5N6PKZWMHPBUVBVKFKERW323NDSLO72A3EJQBCWLXP6Q, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.830207153 (540h11m4.701290199s) [noise=true , address=145.79.227.46:4602 , piece_id=KNU4QZUG2OVV4GXFMEUEHDRSY4DJCIQNU2BXMSHTL6EWDYNR3ZQA, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.830742753 (540h11m4.701825828s) [noise=true , address=45.91.116.8:4048 , piece_id=ID5R5EKJ2ETN6GP4TM4JL4W6NLY2QXW4UHLGCJKPWTVB2BLRMR4A, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.831151041 (540h11m4.702234086s) [noise=true , address=145.79.227.47:4265 , piece_id=RQU4AMAWLQ2LMSN7XTTVQVL2JTALEYAGJ4ETSCD3EE6U6WZY553Q, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.831540642 (540h11m4.702623687s) [noise=true , address=145.79.227.29:4941 , piece_id=PKQ5W6WGYVNVOT3LQLA2G3PPDXC2OE5M6NQFZCK2MIWHCNBTWFDA, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.832020166 (540h11m4.703103241s) [noise=true , address=77.90.31.18:4805 , piece_id=VTHYJLFW5QBBIHOQKOK7RZ4PKZAD7U333XRVM3MJENIUXMJ4C5PQ, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.832408224 (540h11m4.70349127s) [noise=true , address=145.79.227.43:4157 , piece_id=5KWGZV27L6MOGAUWM7MFMSLMAS245IKHQEFF7YICMWLK4V3RYL2A, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.834904358 (540h11m4.705987393s) [noise=true , address=45.91.116.12:4226 , piece_id=G5G56PCQEKBRSJTDKO7KLEUPOO5FQ2TYUCGGPRRN6AKQHMRLYGBQ, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.836744963 (540h11m4.707828008s) [noise=true , address=45.91.116.9:4262 , piece_id=OWP3CND5QXWSJOUSQDKKIME6GGZMI6MAIY4JO2TZ2L7LVO5FVFUQ, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.838330832 (540h11m4.709413877s) [noise=true , address=145.79.227.12:4765 , piece_id=QTASJZDAHZJLVQ5LKWATYOINATQBNYLQ4PUSZZTQVZZMYDW7RMBA, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.839880453 (540h11m4.710963498s) [noise=true , address=77.90.31.15:4700 , piece_id=LMZA3DNDRMZXYFR7JOKDKBR6CZ5XMLPM57T7E4NKXD5BRHL4DWPQ, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.840507678 (540h11m4.711590723s) [noise=true , address=145.79.227.6:4258 , piece_id=OXRMNT2G7QQCOLGC5J433ACEOOZ3WYXAG2Y7DB6NSQO2H6CMH24A, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.843020041 (540h11m4.714103096s) [noise=true , address=145.79.227.34:4906 , piece_id=QM3MPDKVENVQQ2UUH5BAVGQJ4QPJ2IDTIC64M32PN73AIPGW2UZQ, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.843662013 (540h11m4.714745058s) [noise=true , address=77.90.31.13:4626 , piece_id=PNBZUM5FZNZLJLMLGKJDSDZXARXXX6JQGMMPCHTSB7MGWBR4RP4A, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:04.847582178 (540h11m4.718665213s) [noise=true , address=45.91.116.4:4937 , piece_id=XNAKVYJD37CBVSAM6C7TBNS65BLXVRK3TQ55KNJXNMNIAAPKIT4A, seg_pos={0 0}, upload=473731]: Uploading piece... Dec 28 03:40:06.696730094 (540h11m6.567813129s) [upload=473731]: Next returned nil segment Dec 28 03:40:06.696752687 (540h11m6.567835732s) [upload=473731]: Waiting for error group managing segments... Dec 28 03:40:07.005030279 (540h11m6.876113334s) [noise=true , address=145.79.227.46:4602 , piece_id=KNU4QZUG2OVV4GXFMEUEHDRSY4DJCIQNU2BXMSHTL6EWDYNR3ZQA, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.006748491 (540h11m6.877831526s) [noise=true , address=77.90.31.18:4805 , piece_id=VTHYJLFW5QBBIHOQKOK7RZ4PKZAD7U333XRVM3MJENIUXMJ4C5PQ, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.014028663 (540h11m6.885111699s) [noise=true , address=145.79.227.42:4123 , piece_id=EY6AQNGX5N6PKZWMHPBUVBVKFKERW323NDSLO72A3EJQBCWLXP6Q, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.073526548 (540h11m6.944609583s) [noise=true , address=145.79.227.12:4765 , piece_id=QTASJZDAHZJLVQ5LKWATYOINATQBNYLQ4PUSZZTQVZZMYDW7RMBA, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.082519081 (540h11m6.953602116s) [noise=true , address=145.79.227.39:4014 , piece_id=RHDZ4USCKOYCC27JSN23TXXPKFLDVLQEL4USCDHBHKFYNLDFZPEA, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.126819235 (540h11m6.99790227s) [noise=true , address=145.79.227.47:4265 , piece_id=RQU4AMAWLQ2LMSN7XTTVQVL2JTALEYAGJ4ETSCD3EE6U6WZY553Q, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.134320329 (540h11m7.005403364s) [noise=true , address=45.91.116.9:4262 , piece_id=OWP3CND5QXWSJOUSQDKKIME6GGZMI6MAIY4JO2TZ2L7LVO5FVFUQ, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.161896949 (540h11m7.032979995s) [noise=true , address=45.91.116.4:4937 , piece_id=XNAKVYJD37CBVSAM6C7TBNS65BLXVRK3TQ55KNJXNMNIAAPKIT4A, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.188850723 (540h11m7.059933768s) [noise=true , address=45.91.116.12:4226 , piece_id=G5G56PCQEKBRSJTDKO7KLEUPOO5FQ2TYUCGGPRRN6AKQHMRLYGBQ, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:07.418014770 (540h11m7.289097815s) [noise=true , address=45.91.116.8:4048 , piece_id=ID5R5EKJ2ETN6GP4TM4JL4W6NLY2QXW4UHLGCJKPWTVB2BLRMR4A, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:08.084970544 (540h11m7.956053589s) [noise=true , address=145.79.227.34:4906 , piece_id=QM3MPDKVENVQQ2UUH5BAVGQJ4QPJ2IDTIC64M32PN73AIPGW2UZQ, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:08.407849179 (540h11m8.278932225s) [noise=true , address=145.79.227.43:4157 , piece_id=5KWGZV27L6MOGAUWM7MFMSLMAS245IKHQEFF7YICMWLK4V3RYL2A, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:08.455354317 (540h11m8.326437362s) [noise=true , address=77.90.31.13:4626 , piece_id=PNBZUM5FZNZLJLMLGKJDSDZXARXXX6JQGMMPCHTSB7MGWBR4RP4A, seg_pos={0 0}, upload=473731]: Done uploading piece. err: <nil> Dec 28 03:40:08.455384785 (540h11m8.32646783s) [seg_pos={0 0}, upload=473731]: Segment reached optimal threshold of 13 pieces. Dec 28 03:40:08.455594144 (540h11m8.326677179s) [noise=true , address=145.79.227.6:4258 , piece_id=OXRMNT2G7QQCOLGC5J433ACEOOZ3WYXAG2Y7DB6NSQO2H6CMH24A, seg_pos={0 0}, upload=473731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: expected piece hash; context canceled; EOF Dec 28 03:40:08.455602951 (540h11m8.326685996s) [noise=true , address=145.79.227.6:4258 , piece_id=OXRMNT2G7QQCOLGC5J433ACEOOZ3WYXAG2Y7DB6NSQO2H6CMH24A, seg_pos={0 0}, upload=473731]: Long tail cancellation due to optimal threshold Dec 28 03:40:08.455877052 (540h11m8.326960097s) [noise=true , address=145.79.227.29:4941 , piece_id=PKQ5W6WGYVNVOT3LQLA2G3PPDXC2OE5M6NQFZCK2MIWHCNBTWFDA, seg_pos={0 0}, upload=473731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 28 03:40:08.455884447 (540h11m8.326967492s) [noise=true , address=145.79.227.29:4941 , piece_id=PKQ5W6WGYVNVOT3LQLA2G3PPDXC2OE5M6NQFZCK2MIWHCNBTWFDA, seg_pos={0 0}, upload=473731]: Long tail cancellation due to optimal threshold Dec 28 03:40:08.456034462 (540h11m8.327117497s) [noise=true , address=77.90.31.15:4700 , piece_id=LMZA3DNDRMZXYFR7JOKDKBR6CZ5XMLPM57T7E4NKXD5BRHL4DWPQ, seg_pos={0 0}, upload=473731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 03:40:08.456040894 (540h11m8.327123929s) [noise=true , address=77.90.31.15:4700 , piece_id=LMZA3DNDRMZXYFR7JOKDKBR6CZ5XMLPM57T7E4NKXD5BRHL4DWPQ, seg_pos={0 0}, upload=473731]: Long tail cancellation due to optimal threshold Dec 28 03:40:08.456065792 (540h11m8.327148827s) [seg_pos={0 0}, upload=473731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:40:08.456155122 (540h11m8.327238177s) [upload=473731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:40:08.456163679 (540h11m8.327246724s) [upload=473731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:40:08.741353046 (540h11m8.612436091s) [upload=473731]: Done uploading segments... Dec 28 03:40:08.741377853 (540h11m8.612460898s) [upload=473731]: Upload finished. err: <nil> Duration: 30.004000847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 38.049413308s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:40:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 38.049429629s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X....X...........X.....XX.....X..X.X....X.XXXX....X.X.X...] duration: 3.546766794s trace: https://jaeger.dev.storj.io/trace/e58e6d8f62cda28 FAIL (2025-12-28 03:40:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546798675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X....X...........X.....XX.....X..X.X....X.XXXX....X.X.X..] duration: 3.678319796s trace: https://jaeger.dev.storj.io/trace/e58e6d8f62cda28 FAIL (2025-12-28 03:39:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.678340655s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 39.1495018s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:39:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 39.149520155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXXX] duration: 30.005893441s trace: https://jaeger.dev.storj.io/trace/5c40ce8dd55a9448 FAIL (2025-12-28 03:39:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4.132642681 (540h10m14.003725716s) [upload=473717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:39:14.375246740 (540h10m14.246329785s) [seg_pos={0 0}, upload=473717]: Begin upload segment... Dec 28 03:39:14.375315200 (540h10m14.246398235s) [seg_pos={0 0}, upload=473717]: Done begin upload segment. Dec 28 03:39:14.375373171 (540h10m14.246456206s) [noise=true , address=145.79.227.16:4909 , piece_id=AK74BG3V5QNO75DCTHJLFN5K7FCHW2UX6S4RDSCXRHRSE7YV2TJA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.375410472 (540h10m14.246493537s) [noise=true , address=77.90.31.15:4697 , piece_id=3V3AHPLYG3FVU5F42TOQPDEWP5PPJIPD3U5CKUZTLQ7YLAS756XQ, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.376194836 (540h10m14.247277871s) [noise=true , address=145.79.227.42:4123 , piece_id=RWGHYFNQS77BZ2WB224A7AQA2CW75PNTGE635EJPUVZB73HG6E3A, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.376700168 (540h10m14.247783203s) [noise=true , address=145.79.227.15:4873 , piece_id=O74YXZPVTYEAB7PNNUGOQBCXEGYWOW2PTQBWXCFJR4CMXF34KL3Q, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.377248182 (540h10m14.248331217s) [noise=true , address=77.90.31.11:4553 , piece_id=JKXC4OMUPR7LHDP3DYHSBIHBK3TQVADR5BYHTG6TM2YD6Z4W66WQ, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.377626382 (540h10m14.248709417s) [noise=true , address=145.79.227.50:4375 , piece_id=P3AXXGZQFLQQ6F5MURYZQR6ZKHKFNZ6D3E75U3EC3XRAOZYWM6VA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.377970086 (540h10m14.249053121s) [noise=true , address=45.91.116.14:4189 , piece_id=S7V5GXGLDFNJMZZ47RM3AUTQGZX2GBJFIUBSCJGEPBDZAYZBDM5Q, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.378103359 (540h10m14.249186404s) [noise=true , address=77.90.31.19:4842 , piece_id=IEFD2HB7KZDTGKUQWR33WWORDYHAW7BEELCQD5OF4QVRPA62VHYA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.378318480 (540h10m14.249401515s) [noise=true , address=145.79.227.46:4602 , piece_id=UVHL4EMIKBGXIK27V4KJC5567YYP35UKGYJYWOWUINFZOTJVDDPQ, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.378457865 (540h10m14.2495409s) [noise=true , address=145.79.227.14:4837 , piece_id=RKCBHPFAFZYCASY2JRFR3H64OQOOD2SMCSWUXZY755O5YEQ7XMMA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.378723881 (540h10m14.249806916s) [noise=true , address=145.79.227.51:4409 , piece_id=J3ZA227IWGCVYZXDSZMFHIPVBJGQVJAFRADHMUX3J37AEFUPQMAA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.378963628 (540h10m14.250046673s) [noise=true , address=45.91.116.11:4120 , piece_id=6BJLFUVWI4LXM36UOESNSTPFRZQI7W7XHDXXFFTERPG2V623MM7Q, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.379070822 (540h10m14.250153867s) [noise=true , address=45.91.116.10:4081 , piece_id=A76VWTPAHNBWHDPQJJYW5HNRUYIQMNQ4D5AE5K36XQBFTBDJ5ZIA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.379318734 (540h10m14.250401769s) [noise=true , address=145.79.227.36:4519 , piece_id=NW2N7U2UXTT6KGHZPW53P5SVVNSGQ5ALYAWDYOVAO2HPZU4UNERA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.379404327 (540h10m14.250487362s) [noise=true , address=145.79.227.10:4402 , piece_id=RQZLS25RHTURE2U6NYPMNIL2O6PT6X4E5WZJFKFVANYKBZXOBEHA, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:14.380003618 (540h10m14.251086653s) [noise=true , address=145.79.227.6:4258 , piece_id=Z5FOR25NW7XN4UTBAYAY5WAUICTTM77LGUR3W7KVCA6CQOFO7KRQ, seg_pos={0 0}, upload=473717]: Uploading piece... Dec 28 03:39:16.086281897 (540h10m15.957364932s) [upload=473717]: Next returned nil segment Dec 28 03:39:16.086295163 (540h10m15.957378198s) [upload=473717]: Waiting for error group managing segments... Dec 28 03:39:16.380055962 (540h10m16.251138997s) [noise=true , address=145.79.227.10:4402 , piece_id=RQZLS25RHTURE2U6NYPMNIL2O6PT6X4E5WZJFKFVANYKBZXOBEHA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.384614412 (540h10m16.255697447s) [noise=true , address=145.79.227.14:4837 , piece_id=RKCBHPFAFZYCASY2JRFR3H64OQOOD2SMCSWUXZY755O5YEQ7XMMA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.391294643 (540h10m16.262377688s) [noise=true , address=145.79.227.50:4375 , piece_id=P3AXXGZQFLQQ6F5MURYZQR6ZKHKFNZ6D3E75U3EC3XRAOZYWM6VA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.393442883 (540h10m16.264525918s) [noise=true , address=145.79.227.51:4409 , piece_id=J3ZA227IWGCVYZXDSZMFHIPVBJGQVJAFRADHMUX3J37AEFUPQMAA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.401487200 (540h10m16.272570235s) [noise=true , address=145.79.227.36:4519 , piece_id=NW2N7U2UXTT6KGHZPW53P5SVVNSGQ5ALYAWDYOVAO2HPZU4UNERA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.402067826 (540h10m16.273150871s) [noise=true , address=145.79.227.42:4123 , piece_id=RWGHYFNQS77BZ2WB224A7AQA2CW75PNTGE635EJPUVZB73HG6E3A, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.408259907 (540h10m16.279342942s) [noise=true , address=145.79.227.16:4909 , piece_id=AK74BG3V5QNO75DCTHJLFN5K7FCHW2UX6S4RDSCXRHRSE7YV2TJA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.410316463 (540h10m16.281399498s) [noise=true , address=145.79.227.6:4258 , piece_id=Z5FOR25NW7XN4UTBAYAY5WAUICTTM77LGUR3W7KVCA6CQOFO7KRQ, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.412266266 (540h10m16.283349311s) [noise=true , address=145.79.227.15:4873 , piece_id=O74YXZPVTYEAB7PNNUGOQBCXEGYWOW2PTQBWXCFJR4CMXF34KL3Q, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.417035627 (540h10m16.288118673s) [noise=true , address=145.79.227.46:4602 , piece_id=UVHL4EMIKBGXIK27V4KJC5567YYP35UKGYJYWOWUINFZOTJVDDPQ, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.645355417 (540h10m16.516438452s) [noise=true , address=45.91.116.14:4189 , piece_id=S7V5GXGLDFNJMZZ47RM3AUTQGZX2GBJFIUBSCJGEPBDZAYZBDM5Q, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.657229788 (540h10m16.528312853s) [noise=true , address=45.91.116.11:4120 , piece_id=6BJLFUVWI4LXM36UOESNSTPFRZQI7W7XHDXXFFTERPG2V623MM7Q, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.757033791 (540h10m16.628116826s) [noise=true , address=45.91.116.10:4081 , piece_id=A76VWTPAHNBWHDPQJJYW5HNRUYIQMNQ4D5AE5K36XQBFTBDJ5ZIA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: <nil> Dec 28 03:39:16.757054901 (540h10m16.628137936s) [seg_pos={0 0}, upload=473717]: Segment reached optimal threshold of 13 pieces. Dec 28 03:39:16.757210748 (540h10m16.628293784s) [noise=true , address=77.90.31.11:4553 , piece_id=JKXC4OMUPR7LHDP3DYHSBIHBK3TQVADR5BYHTG6TM2YD6Z4W66WQ, seg_pos={0 0}, upload=473717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:58724->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:58724->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58724->77.90.31.11:4553: use of closed network connection Dec 28 03:39:16.757216199 (540h10m16.628299234s) [noise=true , address=77.90.31.11:4553 , piece_id=JKXC4OMUPR7LHDP3DYHSBIHBK3TQVADR5BYHTG6TM2YD6Z4W66WQ, seg_pos={0 0}, upload=473717]: Long tail cancellation due to optimal threshold Dec 28 03:39:16.757270932 (540h10m16.628353987s) [noise=true , address=77.90.31.19:4842 , piece_id=IEFD2HB7KZDTGKUQWR33WWORDYHAW7BEELCQD5OF4QVRPA62VHYA, seg_pos={0 0}, upload=473717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:58864->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:58864->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58864->77.90.31.19:4842: use of closed network connection Dec 28 03:39:16.757280702 (540h10m16.628363747s) [noise=true , address=77.90.31.19:4842 , piece_id=IEFD2HB7KZDTGKUQWR33WWORDYHAW7BEELCQD5OF4QVRPA62VHYA, seg_pos={0 0}, upload=473717]: Long tail cancellation due to optimal threshold Dec 28 03:39:16.757422852 (540h10m16.628505897s) [noise=true , address=77.90.31.15:4697 , piece_id=3V3AHPLYG3FVU5F42TOQPDEWP5PPJIPD3U5CKUZTLQ7YLAS756XQ, seg_pos={0 0}, upload=473717]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:52188->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:52188->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52188->77.90.31.15:4697: use of closed network connection Dec 28 03:39:16.757430376 (540h10m16.628513411s) [noise=true , address=77.90.31.15:4697 , piece_id=3V3AHPLYG3FVU5F42TOQPDEWP5PPJIPD3U5CKUZTLQ7YLAS756XQ, seg_pos={0 0}, upload=473717]: Long tail cancellation due to optimal threshold Dec 28 03:39:16.757449532 (540h10m16.628532567s) [seg_pos={0 0}, upload=473717]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:39:16.757521609 (540h10m16.628604655s) [upload=473717]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:39:16.757527912 (540h10m16.628610947s) [upload=473717]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:39:17.052973259 (540h10m16.924056324s) [upload=473717]: Done uploading segments... Dec 28 03:39:17.052992075 (540h10m16.92407511s) [upload=473717]: Upload finished. err: <nil> Duration:* 30.005917947s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [....X....X...........X.....XX.....X..X.X....X.XXXX....X.X.X.] duration: 3.62117863s trace: https://jaeger.dev.storj.io/trace/e58e6d8f62cda28 FAIL (2025-12-28 03:39:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621203437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXXX] duration: 30.00310109s trace: https://jaeger.dev.storj.io/trace/5ee81f2d3f95d61b FAIL (2025-12-28 03:39:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: QXRA6VVNQJAFPHKOMKGRU6FCNKVRGHJ5V4QU3A76A6H6JL4KZJXQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: SIQWB4DV3P7CYAWJ54HOXDVBI4KWX34MHTT4KUKDS5L3DJPFIYSA) context deadline exceeded ======LOGS====== Dec 28 03:38:39.306692138 (540h9m39.177775173s) [upload=473710]: Starting upload Dec 28 03:38:39.306771408 (540h9m39.177854453s) [upload=473710]: Done starting upload Dec 28 03:38:39.306926464 (540h9m39.178009509s) [upload=473710]: Uploading segments... Dec 28 03:38:39.307093773 (540h9m39.178176808s) [upload=473710]: Got next segment. Inline: false Dec 28 03:38:39.307151953 (540h9m39.178235008s) [upload=473710]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:38:39.307157584 (540h9m39.178240619s) [upload=473710]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:38:39.551783962 (540h9m39.422867017s) [seg_pos={0 0}, upload=473710]: Begin upload segment... Dec 28 03:38:39.551837253 (540h9m39.422920298s) [seg_pos={0 0}, upload=473710]: Done begin upload segment. Dec 28 03:38:39.551863403 (540h9m39.422946438s) [noise=true , address=77.90.31.15:4700 , piece_id=6L3UBPF262SMI7VQYD6DXX7PKT4CTRYIXHREOZ3XIM5PEQFHLB7A, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.551964716 (540h9m39.423047751s) [noise=true , address=77.90.31.19:4841 , piece_id=SIQWB4DV3P7CYAWJ54HOXDVBI4KWX34MHTT4KUKDS5L3DJPFIYSA, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.552558757 (540h9m39.423641792s) [noise=true , address=77.90.31.18:4805 , piece_id=OE4NCYQCMCMTWLBCVX27G6YDQ5DJ3UMWDBNIOJG377HSBCERZSIQ, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.552733961 (540h9m39.423816996s) [noise=true , address=45.91.116.12:4228 , piece_id=IWLE5HBSO2CVOASDUNB6GJPM2WF4IO7OYWUVUKPBHE4O3UMMBX7A, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.552968508 (540h9m39.424051543s) [noise=true , address=45.91.116.11:4117 , piece_id=BMOWUT5RLDTLDK4RNEKN25LWISZSPEF2SLKX5L7K7FNVSHE5X45Q, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.553015807 (540h9m39.424098853s) [noise=true , address=145.79.227.10:4402 , piece_id=IA7GYLEG4LL44FTARIO74BQTO5P5SN3JOO6FX62G3EBYQZHZSFFQ, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.553271805 (540h9m39.42435484s) [noise=true , address=145.79.227.47:4265 , piece_id=RXSG3ED7SFNG3WEOVTFHDSOBGKILZQ75BH5BPC6USFVHX3CIOCUQ, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.553365583 (540h9m39.424448618s) [noise=true , address=45.91.116.8:4046 , piece_id=4TNVFFT7SD33H3LOFK4JFRNPVJ4UMY76HBCS3GIREOIJA4TCPX3Q, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.553543612 (540h9m39.424626647s) [noise=true , address=145.79.227.22:4511 , piece_id=KRPG44GOJSJPI4BMODT7SX4POPA4G5PHXJYB2Y76QVPUDABVS5YQ, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.553743433 (540h9m39.424826478s) [noise=true , address=45.91.116.6:4975 , piece_id=ZSIEMBCOQRBRBOJ7VKSEOKDGI7GYNW4UWD4CMWLRM3JCA4ENT7BA, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.553851889 (540h9m39.424934935s) [noise=true , address=145.79.227.5:4223 , piece_id=TMTZ2ID2M5TZW6M4H2MTZWF2UNNEVAPNBC7WQI64T7PZKRZZ6RXQ, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.554155818 (540h9m39.425238884s) [noise=true , address=145.79.227.19:4077 , piece_id=4TR2WXBFN55R7X7RPSTBHAFSRCBLSLDFDM3O4QM5HVV77OG2S43Q, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.554175766 (540h9m39.425258801s) [noise=true , address=145.79.227.14:4837 , piece_id=FIZ3LLVPLZYXQX4AUHTQDBW4PGZAKEDBYXY4DSRYTRVGKZLYXC2Q, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.554458324 (540h9m39.425541359s) [noise=true , address=145.79.227.11:4729 , piece_id=UHDXHGS722INJCMP3MI4XE5UG2MGBKHGCEH2W4HAEJLJEGR5S6LA, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.554528888 (540h9m39.425611923s) [noise=true , address=45.91.116.4:4937 , piece_id=LC3236OEKZ3VE2TQNKGRDJKLRONQO7IQ7V2SO54D5XAVAGKBN6SA, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:39.555121306 (540h9m39.426204341s) [noise=true , address=45.91.116.13:4153 , piece_id=QXRA6VVNQJAFPHKOMKGRU6FCNKVRGHJ5V4QU3A76A6H6JL4KZJXQ, seg_pos={0 0}, upload=473710]: Uploading piece... Dec 28 03:38:41.296470168 (540h9m41.167553213s) [upload=473710]: Next returned nil segment Dec 28 03:38:41.296486789 (540h9m41.167569824s) [upload=473710]: Waiting for error group managing segments... Dec 28 03:38:41.577387220 (540h9m41.448470256s) [noise=true , address=145.79.227.19:4077 , piece_id=4TR2WXBFN55R7X7RPSTBHAFSRCBLSLDFDM3O4QM5HVV77OG2S43Q, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.579714131 (540h9m41.450797167s) [noise=true , address=145.79.227.11:4729 , piece_id=UHDXHGS722INJCMP3MI4XE5UG2MGBKHGCEH2W4HAEJLJEGR5S6LA, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.609586283 (540h9m41.480669318s) [noise=true , address=145.79.227.14:4837 , piece_id=FIZ3LLVPLZYXQX4AUHTQDBW4PGZAKEDBYXY4DSRYTRVGKZLYXC2Q, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.613399735 (540h9m41.48448278s) [noise=true , address=145.79.227.5:4223 , piece_id=TMTZ2ID2M5TZW6M4H2MTZWF2UNNEVAPNBC7WQI64T7PZKRZZ6RXQ, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.618349420 (540h9m41.489432455s) [noise=true , address=145.79.227.10:4402 , piece_id=IA7GYLEG4LL44FTARIO74BQTO5P5SN3JOO6FX62G3EBYQZHZSFFQ, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.620699404 (540h9m41.491782439s) [noise=true , address=145.79.227.22:4511 , piece_id=KRPG44GOJSJPI4BMODT7SX4POPA4G5PHXJYB2Y76QVPUDABVS5YQ, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.739791596 (540h9m41.610874631s) [noise=true , address=77.90.31.15:4700 , piece_id=6L3UBPF262SMI7VQYD6DXX7PKT4CTRYIXHREOZ3XIM5PEQFHLB7A, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.790232333 (540h9m41.661315378s) [noise=true , address=45.91.116.13:4153 , piece_id=QXRA6VVNQJAFPHKOMKGRU6FCNKVRGHJ5V4QU3A76A6H6JL4KZJXQ, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.896394905 (540h9m41.76747794s) [noise=true , address=45.91.116.4:4937 , piece_id=LC3236OEKZ3VE2TQNKGRDJKLRONQO7IQ7V2SO54D5XAVAGKBN6SA, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:41.970068412 (540h9m41.841151457s) [noise=true , address=145.79.227.47:4265 , piece_id=RXSG3ED7SFNG3WEOVTFHDSOBGKILZQ75BH5BPC6USFVHX3CIOCUQ, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:42.058656488 (540h9m41.929739533s) [noise=true , address=45.91.116.12:4228 , piece_id=IWLE5HBSO2CVOASDUNB6GJPM2WF4IO7OYWUVUKPBHE4O3UMMBX7A, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:42.168211079 (540h9m42.039294114s) [noise=true , address=77.90.31.19:4841 , piece_id=SIQWB4DV3P7CYAWJ54HOXDVBI4KWX34MHTT4KUKDS5L3DJPFIYSA, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:42.383991600 (540h9m42.255074635s) [noise=true , address=77.90.31.18:4805 , piece_id=OE4NCYQCMCMTWLBCVX27G6YDQ5DJ3UMWDBNIOJG377HSBCERZSIQ, seg_pos={0 0}, upload=473710]: Done uploading piece. err: <nil> Dec 28 03:38:42.384012350 (540h9m42.255095385s) [seg_pos={0 0}, upload=473710]: Segment reached optimal threshold of 13 pieces. Dec 28 03:38:42.384316429 (540h9m42.255399464s) [noise=true , address=45.91.116.6:4975 , piece_id=ZSIEMBCOQRBRBOJ7VKSEOKDGI7GYNW4UWD4CMWLRM3JCA4ENT7BA, seg_pos={0 0}, upload=473710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:34838->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:38:42.384323071 (540h9m42.255406106s) [noise=true , address=45.91.116.6:4975 , piece_id=ZSIEMBCOQRBRBOJ7VKSEOKDGI7GYNW4UWD4CMWLRM3JCA4ENT7BA, seg_pos={0 0}, upload=473710]: Long tail cancellation due to optimal threshold Dec 28 03:38:42.384448701 (540h9m42.255531736s) [noise=true , address=45.91.116.11:4117 , piece_id=BMOWUT5RLDTLDK4RNEKN25LWISZSPEF2SLKX5L7K7FNVSHE5X45Q, seg_pos={0 0}, upload=473710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 03:38:42.384454391 (540h9m42.255537426s) [noise=true , address=45.91.116.11:4117 , piece_id=BMOWUT5RLDTLDK4RNEKN25LWISZSPEF2SLKX5L7K7FNVSHE5X45Q, seg_pos={0 0}, upload=473710]: Long tail cancellation due to optimal threshold Dec 28 03:38:42.384540516 (540h9m42.255623551s) [noise=true , address=45.91.116.8:4046 , piece_id=4TNVFFT7SD33H3LOFK4JFRNPVJ4UMY76HBCS3GIREOIJA4TCPX3Q, seg_pos={0 0}, upload=473710]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: read tcp 188.166.225.170:39262->45.91.116.8:4046: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:38:42.384559221 (540h9m42.255642256s) [noise=true , address=45.91.116.8:4046 , piece_id=4TNVFFT7SD33H3LOFK4JFRNPVJ4UMY76HBCS3GIREOIJA4TCPX3Q, seg_pos={0 0}, upload=473710]: Long tail cancellation due to optimal threshold Dec 28 03:38:42.384604858 (540h9m42.255687894s) [seg_pos={0 0}, upload=473710]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:38:42.384725738 (540h9m42.255808773s) [upload=473710]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:38:42.384732892 (540h9m42.255815927s) [upload=473710]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:38:42.678225080 (540h9m42.549308135s) [upload=473710]: Done uploading segments... Dec 28 03:38:42.678248645 (540h9m42.549331691s) [upload=473710]: Upload finished. err: <nil> Duration: 30.003124314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 59.568987458s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:39:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 59.569006324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXXX] duration: 30.002627908s trace: https://jaeger.dev.storj.io/trace/64b035b619d22cf8 FAIL (2025-12-28 03:38:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: RPNKUCPVKK62SKSPMZKZQ6QVAX2VY5SZQTZPWWLSTTMRWITCEVFA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 3N4LGHIXK63CVTWDBJ7TIWPWQFCH64IQTXUZ7ATRKERSK3BYUVPQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: TQXIAEP3EWPM7TKXW3XD37ATUARQ527WNJDA7NJ6ZB4S4NEAYEOQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NSRTQPSLT6YI4ZJZFBVIF5FNSKDJRLSJNU76QYSUMVMUWGDTU3CQ) context deadline exceeded ======LOGS====== Dec 28 03:37:56.139439635 (540h8m56.01052268s) [upload=473699]: Starting upload Dec 28 03:37:56.139526159 (540h8m56.010609204s) [upload=473699]: Done starting upload Dec 28 03:37:56.139595561 (540h8m56.010678596s) [upload=473699]: Uploading segments... Dec 28 03:37:56.139745558 (540h8m56.010828613s) [upload=473699]: Got next segment. Inline: false Dec 28 03:37:56.139781606 (540h8m56.010864641s) [upload=473699]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:37:56.139786906 (540h8m56.010869941s) [upload=473699]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:37:56.384327462 (540h8m56.255410497s) [seg_pos={0 0}, upload=473699]: Begin upload segment... Dec 28 03:37:56.384378489 (540h8m56.255461524s) [seg_pos={0 0}, upload=473699]: Done begin upload segment. Dec 28 03:37:56.384415499 (540h8m56.255498544s) [noise=true , address=45.91.116.7:4012 , piece_id=NSRTQPSLT6YI4ZJZFBVIF5FNSKDJRLSJNU76QYSUMVMUWGDTU3CQ, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.384462840 (540h8m56.255545885s) [noise=true , address=145.79.227.19:4078 , piece_id=CRDDN2PNH4VJ66QSTP5XG6V4EHEQRESUQDGZQZN55ADMWZRN3X4A, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.384581345 (540h8m56.25566438s) [noise=true , address=77.90.31.17:4770 , piece_id=RPNKUCPVKK62SKSPMZKZQ6QVAX2VY5SZQTZPWWLSTTMRWITCEVFA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.384979985 (540h8m56.25606302s) [noise=true , address=45.91.116.10:4083 , piece_id=3N4LGHIXK63CVTWDBJ7TIWPWQFCH64IQTXUZ7ATRKERSK3BYUVPQ, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.385271359 (540h8m56.256354394s) [noise=true , address=145.79.227.45:4229 , piece_id=HWUMR2TRJQ3PCAOOK5GZL4DQYOJAN2DU37UCU2NCK5ODEDGNSABA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.385625243 (540h8m56.256708278s) [noise=true , address=45.91.116.5:4297 , piece_id=CS3RKKFQ73PZK7KSJBMFSLT7RIBDVR6A3D45UPUNTNRI6C5BLFSA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.385937818 (540h8m56.257020853s) [noise=true , address=145.79.227.46:4601 , piece_id=UFC6PKLFBIAGTATGXEUAWYPGU2UQ7RQN5EJSXEANGUTFNGSKROLA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.386330495 (540h8m56.25741353s) [noise=true , address=45.91.116.4:4939 , piece_id=AE4NVCNRQKOFSIY64ZXVASACWAZF5R3RR6LYQLNTM4A7OQ5ACM2Q, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.387999053 (540h8m56.259082098s) [noise=true , address=145.79.227.38:4978 , piece_id=VBAENHTJGFEPV52HBWBW6JOKXKVHFDKECXWASMNTURJB4YIQSZZA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.389326500 (540h8m56.260409545s) [noise=true , address=77.90.31.11:4555 , piece_id=YUJWWSODBCABVC6C77WB7WF5QFFBEOQLYNL6WS3CRMGGYFBMYJUA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.390793264 (540h8m56.261876299s) [noise=true , address=45.91.116.9:4264 , piece_id=EMT5AXY7SQ55CVHPGX2W77QDRALDMYJ6QUBJXUHZT5AKTHC3O4QA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.391291112 (540h8m56.262374147s) [noise=true , address=145.79.227.42:4121 , piece_id=FX4JNJBSNLDHTZLYPE4XVLN5EGK7RPVIBTHS46L3ZTU4D6LRYCBA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.391963943 (540h8m56.263046978s) [noise=true , address=145.79.227.32:4835 , piece_id=MCXKA6JNBNLJO64QLCML6QHJHHE5NKDBLBQQ6YYGIAL54VYWLTAA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.393047646 (540h8m56.264130681s) [noise=true , address=145.79.227.11:4729 , piece_id=NLSUWU3NZD6IH236G7RUPPUWSUSLEQQZIRPRSSFCGHLHBV4ARHTQ, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.394369744 (540h8m56.265452789s) [noise=true , address=145.79.227.12:4767 , piece_id=N3RTPC2MLBQN7D2YR2L6FSTGALPRSBLFLAEYSDEVCQCX4XB4M3EA, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:56.395797983 (540h8m56.266881028s) [noise=true , address=77.90.31.10:4518 , piece_id=TQXIAEP3EWPM7TKXW3XD37ATUARQ527WNJDA7NJ6ZB4S4NEAYEOQ, seg_pos={0 0}, upload=473699]: Uploading piece... Dec 28 03:37:58.243910600 (540h8m58.114993645s) [upload=473699]: Next returned nil segment Dec 28 03:37:58.243923404 (540h8m58.115006439s) [upload=473699]: Waiting for error group managing segments... Dec 28 03:37:58.552342906 (540h8m58.423425941s) [noise=true , address=145.79.227.32:4835 , piece_id=MCXKA6JNBNLJO64QLCML6QHJHHE5NKDBLBQQ6YYGIAL54VYWLTAA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.569856115 (540h8m58.44093915s) [noise=true , address=145.79.227.19:4078 , piece_id=CRDDN2PNH4VJ66QSTP5XG6V4EHEQRESUQDGZQZN55ADMWZRN3X4A, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.572426880 (540h8m58.443509915s) [noise=true , address=145.79.227.11:4729 , piece_id=NLSUWU3NZD6IH236G7RUPPUWSUSLEQQZIRPRSSFCGHLHBV4ARHTQ, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.573172370 (540h8m58.444255415s) [noise=true , address=145.79.227.38:4978 , piece_id=VBAENHTJGFEPV52HBWBW6JOKXKVHFDKECXWASMNTURJB4YIQSZZA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.587293278 (540h8m58.458376323s) [noise=true , address=77.90.31.17:4770 , piece_id=RPNKUCPVKK62SKSPMZKZQ6QVAX2VY5SZQTZPWWLSTTMRWITCEVFA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.646962589 (540h8m58.518045634s) [noise=true , address=45.91.116.4:4939 , piece_id=AE4NVCNRQKOFSIY64ZXVASACWAZF5R3RR6LYQLNTM4A7OQ5ACM2Q, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.660593685 (540h8m58.53167672s) [noise=true , address=77.90.31.10:4518 , piece_id=TQXIAEP3EWPM7TKXW3XD37ATUARQ527WNJDA7NJ6ZB4S4NEAYEOQ, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.697455876 (540h8m58.568538911s) [noise=true , address=145.79.227.45:4229 , piece_id=HWUMR2TRJQ3PCAOOK5GZL4DQYOJAN2DU37UCU2NCK5ODEDGNSABA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.717437715 (540h8m58.58852076s) [noise=true , address=45.91.116.9:4264 , piece_id=EMT5AXY7SQ55CVHPGX2W77QDRALDMYJ6QUBJXUHZT5AKTHC3O4QA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:58.841441469 (540h8m58.712524504s) [noise=true , address=45.91.116.5:4297 , piece_id=CS3RKKFQ73PZK7KSJBMFSLT7RIBDVR6A3D45UPUNTNRI6C5BLFSA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:59.321073849 (540h8m59.192156884s) [noise=true , address=45.91.116.10:4083 , piece_id=3N4LGHIXK63CVTWDBJ7TIWPWQFCH64IQTXUZ7ATRKERSK3BYUVPQ, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:59.347274809 (540h8m59.218357844s) [noise=true , address=145.79.227.12:4767 , piece_id=N3RTPC2MLBQN7D2YR2L6FSTGALPRSBLFLAEYSDEVCQCX4XB4M3EA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:59.348139656 (540h8m59.219222691s) [noise=true , address=45.91.116.7:4012 , piece_id=NSRTQPSLT6YI4ZJZFBVIF5FNSKDJRLSJNU76QYSUMVMUWGDTU3CQ, seg_pos={0 0}, upload=473699]: Done uploading piece. err: <nil> Dec 28 03:37:59.348149436 (540h8m59.219232471s) [seg_pos={0 0}, upload=473699]: Segment reached optimal threshold of 13 pieces. Dec 28 03:37:59.348358554 (540h8m59.219441589s) [noise=true , address=145.79.227.46:4601 , piece_id=UFC6PKLFBIAGTATGXEUAWYPGU2UQ7RQN5EJSXEANGUTFNGSKROLA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 28 03:37:59.348366479 (540h8m59.219449514s) [noise=true , address=145.79.227.46:4601 , piece_id=UFC6PKLFBIAGTATGXEUAWYPGU2UQ7RQN5EJSXEANGUTFNGSKROLA, seg_pos={0 0}, upload=473699]: Long tail cancellation due to optimal threshold Dec 28 03:37:59.348662241 (540h8m59.219745276s) [noise=true , address=145.79.227.42:4121 , piece_id=FX4JNJBSNLDHTZLYPE4XVLN5EGK7RPVIBTHS46L3ZTU4D6LRYCBA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 28 03:37:59.348668703 (540h8m59.219751748s) [noise=true , address=145.79.227.42:4121 , piece_id=FX4JNJBSNLDHTZLYPE4XVLN5EGK7RPVIBTHS46L3ZTU4D6LRYCBA, seg_pos={0 0}, upload=473699]: Long tail cancellation due to optimal threshold Dec 28 03:37:59.348818780 (540h8m59.219901815s) [noise=true , address=77.90.31.11:4555 , piece_id=YUJWWSODBCABVC6C77WB7WF5QFFBEOQLYNL6WS3CRMGGYFBMYJUA, seg_pos={0 0}, upload=473699]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38594->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:37:59.348825222 (540h8m59.219908257s) [noise=true , address=77.90.31.11:4555 , piece_id=YUJWWSODBCABVC6C77WB7WF5QFFBEOQLYNL6WS3CRMGGYFBMYJUA, seg_pos={0 0}, upload=473699]: Long tail cancellation due to optimal threshold Dec 28 03:37:59.348841132 (540h8m59.219924167s) [seg_pos={0 0}, upload=473699]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:37:59.348905274 (540h8m59.219988319s) [upload=473699]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:37:59.348910604 (540h8m59.219993639s) [upload=473699]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:37:59.635599748 (540h8m59.506682793s) [upload=473699]: Done uploading segments... Dec 28 03:37:59.635618344 (540h8m59.506701379s) [upload=473699]: Upload finished. err: <nil> Duration: 30.002640491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.....X....X...........X.....XX.....X..X.X....X.XXXX....X.X.X] duration: 5.001653324s trace: https://jaeger.dev.storj.io/trace/e58e6d8f62cda28 FAIL (2025-12-28 03:38:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001715111s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 44.877023738s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:38:09 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 44.877078152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XXX] duration: 30.002774573s trace: https://jaeger.dev.storj.io/trace/49fdef1211cc3ebf FAIL (2025-12-28 03:37:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: LPFYSNXRNJKJDGTQRP5JSWGJRLGSEU7OFPSHROTKWHBA32ARDKEQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: TJIKEXBUR5DFQOGOHCLRWBLVQ2RRXKW3M5L5C2O6FSBAUDVCKWUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: AAVBF6DKQJQT5FXA6M64UPDE3XCEL3F3LOEBY7UN2D2JN7OSN4OA) context deadline exceeded; piecestore: (Node ID: 12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, Piece ID: A7ZYSR7GH7SYWAN54GFXQEKNF2II4Q7QI3PFDSBD5OGX4PCFTSJA) context deadline exceeded ======LOGS====== chRequestItem_ObjectBegin Dec 28 03:37:25.917573597 (540h8m25.788656642s) [upload=473691]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:37:26.163187737 (540h8m26.034270772s) [seg_pos={0 0}, upload=473691]: Begin upload segment... Dec 28 03:37:26.167884611 (540h8m26.038967656s) [seg_pos={0 0}, upload=473691]: Done begin upload segment. Dec 28 03:37:26.167954434 (540h8m26.039037469s) [noise=true , address=145.79.227.9:4365 , piece_id=NIZUZFL67GSVTUEKQLWVNYRBEGRTPYSNAL7CWKHQWIRVDJTDIESA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.171880289 (540h8m26.042963324s) [noise=true , address=145.79.227.42:4121 , piece_id=5IH6EV5ITGDCU3P4JBJ4U2ZPRKGPAJKXBKSZPU3GUNSRTYHIWAHQ, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.172490871 (540h8m26.043573906s) [noise=true , address=145.79.227.37:4567 , piece_id=D3L4SMOOPAVZJO5FTHK6S36SO2UDITK7CD2EGSP6MGITFXST73CA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.173027023 (540h8m26.044110058s) [noise=true , address=145.79.227.45:4230 , piece_id=A7ZYSR7GH7SYWAN54GFXQEKNF2II4Q7QI3PFDSBD5OGX4PCFTSJA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.173502177 (540h8m26.044585212s) [noise=true , address=77.90.31.11:4555 , piece_id=4N45OJUNME2CFGFSFZOLL2OMBXROHBLNJ4I53IITUTTC3JGSFF2A, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.173570808 (540h8m26.044653853s) [noise=true , address=45.91.116.5:4298 , piece_id=HV727PHW75JYNVFU4HHPAYEKZLV4XPFSFD6Y6MZM4OYPJ36SMC3A, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.174248859 (540h8m26.045331904s) [noise=true , address=45.91.116.13:4156 , piece_id=T3PJJSF6POO4VMCGVORQTNHKOUDUIOHITDQ4VYHV77AGG3L7TBNQ, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.174336386 (540h8m26.045419421s) [noise=true , address=145.79.227.15:4873 , piece_id=7JVU7L3L5AW5DUMEK4WGXS2J3NXDRLDO2U3UZHBRPW2X3ZYGPELA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.174783637 (540h8m26.045866672s) [noise=true , address=45.91.116.8:4048 , piece_id=LPFYSNXRNJKJDGTQRP5JSWGJRLGSEU7OFPSHROTKWHBA32ARDKEQ, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.175322995 (540h8m26.04640603s) [noise=true , address=145.79.227.7:4293 , piece_id=TJIKEXBUR5DFQOGOHCLRWBLVQ2RRXKW3M5L5C2O6FSBAUDVCKWUA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.175498709 (540h8m26.046581755s) [noise=true , address=145.79.227.18:4007 , piece_id=5G2DWZ4NXKORAZUP7TBD6X2FJ5QYJXI3PLPYIJIKXFUTNB6P7M4Q, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.176022617 (540h8m26.047105652s) [noise=true , address=45.91.116.14:4191 , piece_id=VGOXEF7JRPOTZ5E3V3VNBCJV5LRCILDZ2SMMHSGTHG436LCO5NSA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.176203191 (540h8m26.047286226s) [noise=true , address=145.79.227.17:4970 , piece_id=KF6FD4ZXWRUIVJ4XQDDV32RYVURDEH5VACB3OFCX2ZI4J6UPVUNQ, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.176550793 (540h8m26.047633828s) [noise=true , address=45.91.116.6:4974 , piece_id=AAVBF6DKQJQT5FXA6M64UPDE3XCEL3F3LOEBY7UN2D2JN7OSN4OA, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.176924084 (540h8m26.048007119s) [noise=true , address=145.79.227.32:4835 , piece_id=3CQR4IRCI6B6S7BGVIME7GX7EMOLWBB7SGQEGNIAKUK6JFYNBNHQ, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:26.177214516 (540h8m26.048297551s) [noise=true , address=77.90.31.17:4770 , piece_id=E2ATNXFJ3MSUQJ37QEIKTLAQ5O7IUOA6NPEGADC4VXLRXUURWUQQ, seg_pos={0 0}, upload=473691]: Uploading piece... Dec 28 03:37:28.023216012 (540h8m27.894299047s) [upload=473691]: Next returned nil segment Dec 28 03:37:28.023230049 (540h8m27.894313084s) [upload=473691]: Waiting for error group managing segments... Dec 28 03:37:28.337095982 (540h8m28.208179017s) [noise=true , address=145.79.227.17:4970 , piece_id=KF6FD4ZXWRUIVJ4XQDDV32RYVURDEH5VACB3OFCX2ZI4J6UPVUNQ, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.337883752 (540h8m28.208966787s) [noise=true , address=145.79.227.7:4293 , piece_id=TJIKEXBUR5DFQOGOHCLRWBLVQ2RRXKW3M5L5C2O6FSBAUDVCKWUA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.354088308 (540h8m28.225171343s) [noise=true , address=145.79.227.9:4365 , piece_id=NIZUZFL67GSVTUEKQLWVNYRBEGRTPYSNAL7CWKHQWIRVDJTDIESA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.358274089 (540h8m28.229357124s) [noise=true , address=145.79.227.18:4007 , piece_id=5G2DWZ4NXKORAZUP7TBD6X2FJ5QYJXI3PLPYIJIKXFUTNB6P7M4Q, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.361447581 (540h8m28.232530626s) [noise=true , address=145.79.227.45:4230 , piece_id=A7ZYSR7GH7SYWAN54GFXQEKNF2II4Q7QI3PFDSBD5OGX4PCFTSJA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.446709965 (540h8m28.317793s) [noise=true , address=45.91.116.8:4048 , piece_id=LPFYSNXRNJKJDGTQRP5JSWGJRLGSEU7OFPSHROTKWHBA32ARDKEQ, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.461524274 (540h8m28.332607309s) [noise=true , address=77.90.31.17:4770 , piece_id=E2ATNXFJ3MSUQJ37QEIKTLAQ5O7IUOA6NPEGADC4VXLRXUURWUQQ, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.536129926 (540h8m28.407212971s) [noise=true , address=45.91.116.5:4298 , piece_id=HV727PHW75JYNVFU4HHPAYEKZLV4XPFSFD6Y6MZM4OYPJ36SMC3A, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.572015909 (540h8m28.443098944s) [noise=true , address=45.91.116.14:4191 , piece_id=VGOXEF7JRPOTZ5E3V3VNBCJV5LRCILDZ2SMMHSGTHG436LCO5NSA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.618816224 (540h8m28.489899269s) [noise=true , address=45.91.116.13:4156 , piece_id=T3PJJSF6POO4VMCGVORQTNHKOUDUIOHITDQ4VYHV77AGG3L7TBNQ, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.642492858 (540h8m28.513575893s) [noise=true , address=145.79.227.42:4121 , piece_id=5IH6EV5ITGDCU3P4JBJ4U2ZPRKGPAJKXBKSZPU3GUNSRTYHIWAHQ, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.662777614 (540h8m28.533860659s) [noise=true , address=145.79.227.15:4873 , piece_id=7JVU7L3L5AW5DUMEK4WGXS2J3NXDRLDO2U3UZHBRPW2X3ZYGPELA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.740612164 (540h8m28.611695199s) [noise=true , address=45.91.116.6:4974 , piece_id=AAVBF6DKQJQT5FXA6M64UPDE3XCEL3F3LOEBY7UN2D2JN7OSN4OA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: <nil> Dec 28 03:37:28.740637983 (540h8m28.611721028s) [seg_pos={0 0}, upload=473691]: Segment reached optimal threshold of 13 pieces. Dec 28 03:37:28.740923397 (540h8m28.612006442s) [noise=true , address=77.90.31.11:4555 , piece_id=4N45OJUNME2CFGFSFZOLL2OMBXROHBLNJ4I53IITUTTC3JGSFF2A, seg_pos={0 0}, upload=473691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:53528->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:53528->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53528->77.90.31.11:4555: use of closed network connection Dec 28 03:37:28.740938145 (540h8m28.61202118s) [noise=true , address=77.90.31.11:4555 , piece_id=4N45OJUNME2CFGFSFZOLL2OMBXROHBLNJ4I53IITUTTC3JGSFF2A, seg_pos={0 0}, upload=473691]: Long tail cancellation due to optimal threshold Dec 28 03:37:28.741032484 (540h8m28.612115529s) [noise=true , address=145.79.227.32:4835 , piece_id=3CQR4IRCI6B6S7BGVIME7GX7EMOLWBB7SGQEGNIAKUK6JFYNBNHQ, seg_pos={0 0}, upload=473691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y): protocol: write tcp 188.166.225.170:38130->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:38130->145.79.227.32:4835: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38130->145.79.227.32:4835: use of closed network connection Dec 28 03:37:28.741045460 (540h8m28.612128495s) [noise=true , address=145.79.227.32:4835 , piece_id=3CQR4IRCI6B6S7BGVIME7GX7EMOLWBB7SGQEGNIAKUK6JFYNBNHQ, seg_pos={0 0}, upload=473691]: Long tail cancellation due to optimal threshold Dec 28 03:37:28.741253696 (540h8m28.612336741s) [noise=true , address=145.79.227.37:4567 , piece_id=D3L4SMOOPAVZJO5FTHK6S36SO2UDITK7CD2EGSP6MGITFXST73CA, seg_pos={0 0}, upload=473691]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:48168->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.37:4567: use of closed network connection Dec 28 03:37:28.741263705 (540h8m28.61234676s) [noise=true , address=145.79.227.37:4567 , piece_id=D3L4SMOOPAVZJO5FTHK6S36SO2UDITK7CD2EGSP6MGITFXST73CA, seg_pos={0 0}, upload=473691]: Long tail cancellation due to optimal threshold Dec 28 03:37:28.741302659 (540h8m28.612385704s) [seg_pos={0 0}, upload=473691]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:37:28.741421255 (540h8m28.6125043s) [upload=473691]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:37:28.741428399 (540h8m28.612511444s) [upload=473691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:37:29.039019051 (540h8m28.910102096s) [upload=473691]: Done uploading segments... Dec 28 03:37:29.039168266 (540h8m28.910251301s) [upload=473691]: Upload finished. err: <nil> Duration:* 30.002792398s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [......X....X...........X.....XX.....X..X.X....X.XXXX....X.X.] duration: 3.399010693s trace: https://jaeger.dev.storj.io/trace/55132911f76a95a3 FAIL (2025-12-28 03:37:26 UTC) orbiter warning: context deadline exceeded Duration: 3.399035782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 38.435430857s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:36:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 38.435458891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.XX] duration: 30.002367084s trace: https://jaeger.dev.storj.io/trace/31831a6e047f951a FAIL (2025-12-28 03:36:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HIQ5GYZGM6R7QLHEOTHNCUU5XYRD2AXVC2GMCR64FMBNSTC54ZWA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: V2F45POPJZQEGD52K3ZJOFG7AX34SZPR3OYHQQUB3VDL3KJS455A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: KBK77ABFSGTE3PBWTFGM6FSTS26LJQGQA5B2VFRNBC6CHYRU67OQ) context deadline exceeded ======LOGS====== 75]: Starting upload Dec 28 03:36:15.726722036 (540h7m15.597805081s) [upload=473675]: Done starting upload Dec 28 03:36:15.726918680 (540h7m15.598001715s) [upload=473675]: Uploading segments... Dec 28 03:36:15.726953326 (540h7m15.598036361s) [upload=473675]: Got next segment. Inline: false Dec 28 03:36:15.726975057 (540h7m15.598058092s) [upload=473675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:36:15.726979516 (540h7m15.598062561s) [upload=473675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:36:15.973584063 (540h7m15.844667108s) [seg_pos={0 0}, upload=473675]: Begin upload segment... Dec 28 03:36:15.973643966 (540h7m15.844727001s) [seg_pos={0 0}, upload=473675]: Done begin upload segment. Dec 28 03:36:15.973682349 (540h7m15.844765394s) [noise=true , address=145.79.227.6:4257 , piece_id=7X2KVPMGZ63UBIDDD2SLAZSPLGPYBPSIG6BJIJQZYL6XWNXMDHSA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.973935181 (540h7m15.845018227s) [noise=true , address=77.90.31.18:4808 , piece_id=X3BEUITEQ3D73POHMFGBS3LYLFA5HET52BMM5XFW5SICYQVOP7NA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.975189079 (540h7m15.846272124s) [noise=true , address=145.79.227.19:4079 , piece_id=IYGFCXQH34XEHJC3JIR3LLJ6DKQG3B73NHEMFFKI5TZTCYK7CTWQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.975631862 (540h7m15.846714897s) [noise=true , address=145.79.227.17:4970 , piece_id=MOBEWRM5CBOJYXTJ5JQ6BZYBSCQM76O2Q7AXLU3WMOCYGSTIWSAQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.976074715 (540h7m15.84715776s) [noise=true , address=145.79.227.48:4301 , piece_id=ER2LXUHQBWHFLESYVZQTXI63QIJWDERBPDH2QJFLWD34E7FQ3UAQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.976472142 (540h7m15.847555187s) [noise=true , address=77.90.31.13:4629 , piece_id=FWP5HATNGHLFW5QXTX5GQVMSMBJFS52QN52P3CYZC4YGW6WQGJMA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.976851876 (540h7m15.847934911s) [noise=true , address=145.79.227.9:4365 , piece_id=34RM7GIPJW6SYOXKU5RDQYGRMNZKQ4PDSCFCVWABLT6Q4GC3EZAA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.978959629 (540h7m15.850042674s) [noise=true , address=145.79.227.11:4732 , piece_id=QT2UBMZNZZQQR2UVYOVFMK74V4D6VICQGCO6W75X2LCUDIHD4POQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.979464601 (540h7m15.850547646s) [noise=true , address=77.90.31.14:4663 , piece_id=J3WC5UHMCRDFM4CYE7SGBD3H3XNZWOIW5V5L5PPL3XWB46XC4OTA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.979914046 (540h7m15.850997081s) [noise=true , address=45.91.116.11:4117 , piece_id=HIQ5GYZGM6R7QLHEOTHNCUU5XYRD2AXVC2GMCR64FMBNSTC54ZWA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.980349485 (540h7m15.85143252s) [noise=true , address=77.90.31.17:4770 , piece_id=V2F45POPJZQEGD52K3ZJOFG7AX34SZPR3OYHQQUB3VDL3KJS455A, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.980786317 (540h7m15.851869362s) [noise=true , address=145.79.227.42:4122 , piece_id=EFXCJY7OGSHC5PGCZK5IJHHZYOJD75QU4AIL5TGHLGWOVMZEGEWA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.982523955 (540h7m15.85360699s) [noise=true , address=145.79.227.36:4519 , piece_id=SAMHAONOG46HNHDMT3KXEEBXT2XKRFJRBQGGFTO76S24O2CXK3RQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.983157823 (540h7m15.854240868s) [noise=true , address=145.79.227.12:4765 , piece_id=WY5XF5SGFV4M2NLOL4RQZUL77MPJ3N3RTRYGHMB6RLPSXZIZRECA, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.983609323 (540h7m15.854692368s) [noise=true , address=145.79.227.29:4943 , piece_id=V23UUPDW2EE6AT3UB5P4ASUEDPEW74DHQS7QSXSDULCMEKHWSJNQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:15.984027389 (540h7m15.855110434s) [noise=true , address=77.90.31.10:4518 , piece_id=KBK77ABFSGTE3PBWTFGM6FSTS26LJQGQA5B2VFRNBC6CHYRU67OQ, seg_pos={0 0}, upload=473675]: Uploading piece... Dec 28 03:36:17.699362663 (540h7m17.570445698s) [upload=473675]: Next returned nil segment Dec 28 03:36:17.699375518 (540h7m17.570458564s) [upload=473675]: Waiting for error group managing segments... Dec 28 03:36:17.982437926 (540h7m17.853520961s) [noise=true , address=145.79.227.17:4970 , piece_id=MOBEWRM5CBOJYXTJ5JQ6BZYBSCQM76O2Q7AXLU3WMOCYGSTIWSAQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:17.989240560 (540h7m17.860323595s) [noise=true , address=145.79.227.9:4365 , piece_id=34RM7GIPJW6SYOXKU5RDQYGRMNZKQ4PDSCFCVWABLT6Q4GC3EZAA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.013218588 (540h7m17.884301623s) [noise=true , address=145.79.227.42:4122 , piece_id=EFXCJY7OGSHC5PGCZK5IJHHZYOJD75QU4AIL5TGHLGWOVMZEGEWA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.044658043 (540h7m17.915741088s) [noise=true , address=145.79.227.29:4943 , piece_id=V23UUPDW2EE6AT3UB5P4ASUEDPEW74DHQS7QSXSDULCMEKHWSJNQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.231923248 (540h7m18.103006293s) [noise=true , address=45.91.116.11:4117 , piece_id=HIQ5GYZGM6R7QLHEOTHNCUU5XYRD2AXVC2GMCR64FMBNSTC54ZWA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.279888823 (540h7m18.150971858s) [noise=true , address=77.90.31.17:4770 , piece_id=V2F45POPJZQEGD52K3ZJOFG7AX34SZPR3OYHQQUB3VDL3KJS455A, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.293446518 (540h7m18.164529564s) [noise=true , address=77.90.31.10:4518 , piece_id=KBK77ABFSGTE3PBWTFGM6FSTS26LJQGQA5B2VFRNBC6CHYRU67OQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.297302371 (540h7m18.168385416s) [noise=true , address=77.90.31.13:4629 , piece_id=FWP5HATNGHLFW5QXTX5GQVMSMBJFS52QN52P3CYZC4YGW6WQGJMA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.604369287 (540h7m18.475452322s) [noise=true , address=145.79.227.12:4765 , piece_id=WY5XF5SGFV4M2NLOL4RQZUL77MPJ3N3RTRYGHMB6RLPSXZIZRECA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.806208162 (540h7m18.677291197s) [noise=true , address=145.79.227.6:4257 , piece_id=7X2KVPMGZ63UBIDDD2SLAZSPLGPYBPSIG6BJIJQZYL6XWNXMDHSA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.808390999 (540h7m18.679474044s) [noise=true , address=145.79.227.36:4519 , piece_id=SAMHAONOG46HNHDMT3KXEEBXT2XKRFJRBQGGFTO76S24O2CXK3RQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.905889933 (540h7m18.776972968s) [noise=true , address=145.79.227.11:4732 , piece_id=QT2UBMZNZZQQR2UVYOVFMK74V4D6VICQGCO6W75X2LCUDIHD4POQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.996573821 (540h7m18.867656856s) [noise=true , address=77.90.31.14:4663 , piece_id=J3WC5UHMCRDFM4CYE7SGBD3H3XNZWOIW5V5L5PPL3XWB46XC4OTA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: <nil> Dec 28 03:36:18.996599268 (540h7m18.867682313s) [seg_pos={0 0}, upload=473675]: Segment reached optimal threshold of 13 pieces. Dec 28 03:36:18.996829397 (540h7m18.867912452s) [noise=true , address=145.79.227.48:4301 , piece_id=ER2LXUHQBWHFLESYVZQTXI63QIJWDERBPDH2QJFLWD34E7FQ3UAQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: write tcp 188.166.225.170:41842->145.79.227.48:4301: use of closed network connection; write tcp 188.166.225.170:41842->145.79.227.48:4301: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41842->145.79.227.48:4301: use of closed network connection Dec 28 03:36:18.996839155 (540h7m18.8679222s) [noise=true , address=145.79.227.48:4301 , piece_id=ER2LXUHQBWHFLESYVZQTXI63QIJWDERBPDH2QJFLWD34E7FQ3UAQ, seg_pos={0 0}, upload=473675]: Long tail cancellation due to optimal threshold Dec 28 03:36:18.997311114 (540h7m18.868394159s) [noise=true , address=77.90.31.18:4808 , piece_id=X3BEUITEQ3D73POHMFGBS3LYLFA5HET52BMM5XFW5SICYQVOP7NA, seg_pos={0 0}, upload=473675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:45910->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:36:18.997321614 (540h7m18.868404659s) [noise=true , address=77.90.31.18:4808 , piece_id=X3BEUITEQ3D73POHMFGBS3LYLFA5HET52BMM5XFW5SICYQVOP7NA, seg_pos={0 0}, upload=473675]: Long tail cancellation due to optimal threshold Dec 28 03:36:18.997474605 (540h7m18.86855765s) [noise=true , address=145.79.227.19:4079 , piece_id=IYGFCXQH34XEHJC3JIR3LLJ6DKQG3B73NHEMFFKI5TZTCYK7CTWQ, seg_pos={0 0}, upload=473675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 28 03:36:18.997483573 (540h7m18.868566618s) [noise=true , address=145.79.227.19:4079 , piece_id=IYGFCXQH34XEHJC3JIR3LLJ6DKQG3B73NHEMFFKI5TZTCYK7CTWQ, seg_pos={0 0}, upload=473675]: Long tail cancellation due to optimal threshold Dec 28 03:36:18.997503831 (540h7m18.868586866s) [seg_pos={0 0}, upload=473675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:36:18.997590977 (540h7m18.868674022s) [upload=473675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:36:18.997597049 (540h7m18.868680094s) [upload=473675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:36:19.286743141 (540h7m19.157826196s) [upload=473675]: Done uploading segments... Dec 28 03:36:19.286765954 (540h7m19.157848999s) [upload=473675]: Upload finished. err: <nil> Duration: 30.002380359s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.......X....X...........X.....XX.....X..X.X....X.XXXX....X.X] duration: 5.001289673s trace: https://jaeger.dev.storj.io/trace/55132911f76a95a3 FAIL (2025-12-28 03:36:26 UTC) orbiter warning: context deadline exceeded Duration: 5.001318137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.X] duration: 30.003057154s trace: https://jaeger.dev.storj.io/trace/15791f08e0c45ccb FAIL (2025-12-28 03:36:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:35:41.560221920 (540h6m41.431304995s) [upload=473668]: Starting upload Dec 28 03:35:41.560290150 (540h6m41.431373205s) [upload=473668]: Done starting upload Dec 28 03:35:41.560408034 (540h6m41.431491079s) [upload=473668]: Uploading segments... Dec 28 03:35:41.560508826 (540h6m41.431591871s) [upload=473668]: Got next segment. Inline: false Dec 28 03:35:41.560540346 (540h6m41.431623391s) [upload=473668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:35:41.560544183 (540h6m41.431627218s) [upload=473668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:35:41.810109177 (540h6m41.681192222s) [seg_pos={0 0}, upload=473668]: Begin upload segment... Dec 28 03:35:41.810164402 (540h6m41.681247447s) [seg_pos={0 0}, upload=473668]: Done begin upload segment. Dec 28 03:35:41.810198207 (540h6m41.681281242s) [noise=true , address=145.79.227.50:4373 , piece_id=QWESKU633JUQF6TBTLGLZF3OAMD42ABQYNWI3GX6VMUIMZ2ABTAQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.810288559 (540h6m41.681371604s) [noise=true , address=45.91.116.13:4154 , piece_id=5LCW4SBCHQSVW4VIKAFSDDTXSFTXF5VMPA52C36XWMIMI5JXDEJQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.810954347 (540h6m41.682037392s) [noise=true , address=77.90.31.15:4700 , piece_id=FDNQEC5XRXFJAUVY2ASPXOKPICKMAQXYSFOE7TLJ7EFPY6IQIMOA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.811016385 (540h6m41.68209942s) [noise=true , address=145.79.227.5:4223 , piece_id=STQSOR4RKXJQIFZZQCSH3J4FPTS6CB4PNKGPRGTNM6I7BNSLHYRA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.811389505 (540h6m41.68247255s) [noise=true , address=45.91.116.10:4082 , piece_id=UZBLM5U6V7IJRT4YF7SRYWYGPMZZH3IWFEPECUYPXJZLTLLYKO5A, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.811682123 (540h6m41.682765158s) [noise=true , address=145.79.227.22:4511 , piece_id=KMHLWOXJTLWOYTHEM6HMKBVYRFK6DR7C7M47EGIWRBHDOWGE6UKA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812053189 (540h6m41.683136234s) [noise=true , address=77.90.31.21:4880 , piece_id=WA4AJZYRLSGQ6Y3H43ZK2WPZV7ANQOJ7F6ZD2X53UCVV4WHIEN5Q, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812086602 (540h6m41.683169647s) [noise=true , address=45.91.116.11:4118 , piece_id=GQLZAX5I7VZEVDXURQZZXLPDDUG3K47VOEKDV7N4Z2IT5FF5WAWQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812356096 (540h6m41.683439141s) [noise=true , address=77.90.31.14:4661 , piece_id=FRBSKDNXKZFK7PUBJR25NFZPERRMAZ2LYAU2UGLWCFV2KXMXG4OA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812584140 (540h6m41.683667176s) [noise=true , address=45.91.116.6:4973 , piece_id=2GCZS6MZKB27MR6RX66RVETYOYG4LFOUWJHKE7D2W6AYNHMLQXMQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812690653 (540h6m41.683773698s) [noise=true , address=145.79.227.10:4402 , piece_id=GR34SYSSX2EMFHOX67HAOYQFHSE4QDUC2NSEDUHWYB6S2SZWQ7MA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.812963432 (540h6m41.684046467s) [noise=true , address=45.91.116.5:4299 , piece_id=QYZ7GOUIU6JS3ITOMFNKXP6ZKW6TOHKHIBCDQCEDIWCQ7O3256WA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.813077710 (540h6m41.684160745s) [noise=true , address=77.90.31.16:4735 , piece_id=3U5HCHJ4TSL6OW7K3RD7KM2TGURI33YI4XVC3YWYRLX4VNHBQRHA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.813564948 (540h6m41.684647993s) [noise=true , address=145.79.227.15:4875 , piece_id=UHMC5GLQR24KIPOLQALNDSCUV4DNAM2VDC7IYSLA4UCEDV6BXPJQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.814057345 (540h6m41.68514038s) [noise=true , address=145.79.227.31:4797 , piece_id=6R7XY2V6ZSFMTQNR3NRYQPS6AS6LBQSDYUHXJZTB6BJR6FZFIMNQ, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:41.815179121 (540h6m41.686262166s) [noise=true , address=145.79.227.16:4912 , piece_id=KIU63KH5T67OKWEJE45QZBNH5OZJIWPA6TT2BJFJAEWZ7AOTGKZA, seg_pos={0 0}, upload=473668]: Uploading piece... Dec 28 03:35:43.551664790 (540h6m43.422747825s) [upload=473668]: Next returned nil segment Dec 28 03:35:43.551678857 (540h6m43.422761893s) [upload=473668]: Waiting for error group managing segments... Dec 28 03:35:43.852598538 (540h6m43.723681573s) [noise=true , address=145.79.227.50:4373 , piece_id=QWESKU633JUQF6TBTLGLZF3OAMD42ABQYNWI3GX6VMUIMZ2ABTAQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.893047079 (540h6m43.764130114s) [noise=true , address=145.79.227.31:4797 , piece_id=6R7XY2V6ZSFMTQNR3NRYQPS6AS6LBQSDYUHXJZTB6BJR6FZFIMNQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.898665047 (540h6m43.769748092s) [noise=true , address=145.79.227.15:4875 , piece_id=UHMC5GLQR24KIPOLQALNDSCUV4DNAM2VDC7IYSLA4UCEDV6BXPJQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.918101117 (540h6m43.789184152s) [noise=true , address=145.79.227.16:4912 , piece_id=KIU63KH5T67OKWEJE45QZBNH5OZJIWPA6TT2BJFJAEWZ7AOTGKZA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:43.922833468 (540h6m43.793916503s) [noise=true , address=145.79.227.22:4511 , piece_id=KMHLWOXJTLWOYTHEM6HMKBVYRFK6DR7C7M47EGIWRBHDOWGE6UKA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.052738718 (540h6m43.923821753s) [noise=true , address=77.90.31.15:4700 , piece_id=FDNQEC5XRXFJAUVY2ASPXOKPICKMAQXYSFOE7TLJ7EFPY6IQIMOA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.121321501 (540h6m43.992404547s) [noise=true , address=77.90.31.16:4735 , piece_id=3U5HCHJ4TSL6OW7K3RD7KM2TGURI33YI4XVC3YWYRLX4VNHBQRHA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.129487191 (540h6m44.000570226s) [noise=true , address=77.90.31.14:4661 , piece_id=FRBSKDNXKZFK7PUBJR25NFZPERRMAZ2LYAU2UGLWCFV2KXMXG4OA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.149713595 (540h6m44.02079663s) [noise=true , address=45.91.116.11:4118 , piece_id=GQLZAX5I7VZEVDXURQZZXLPDDUG3K47VOEKDV7N4Z2IT5FF5WAWQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.176376094 (540h6m44.047459139s) [noise=true , address=45.91.116.5:4299 , piece_id=QYZ7GOUIU6JS3ITOMFNKXP6ZKW6TOHKHIBCDQCEDIWCQ7O3256WA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.185595660 (540h6m44.056678705s) [noise=true , address=77.90.31.21:4880 , piece_id=WA4AJZYRLSGQ6Y3H43ZK2WPZV7ANQOJ7F6ZD2X53UCVV4WHIEN5Q, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.413684970 (540h6m44.284768005s) [noise=true , address=45.91.116.10:4082 , piece_id=UZBLM5U6V7IJRT4YF7SRYWYGPMZZH3IWFEPECUYPXJZLTLLYKO5A, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.833075591 (540h6m44.704158626s) [noise=true , address=45.91.116.6:4973 , piece_id=2GCZS6MZKB27MR6RX66RVETYOYG4LFOUWJHKE7D2W6AYNHMLQXMQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: <nil> Dec 28 03:35:44.833094947 (540h6m44.704177992s) [seg_pos={0 0}, upload=473668]: Segment reached optimal threshold of 13 pieces. Dec 28 03:35:44.833270822 (540h6m44.704353857s) [noise=true , address=45.91.116.13:4154 , piece_id=5LCW4SBCHQSVW4VIKAFSDDTXSFTXF5VMPA52C36XWMIMI5JXDEJQ, seg_pos={0 0}, upload=473668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 03:35:44.833276392 (540h6m44.704359427s) [noise=true , address=45.91.116.13:4154 , piece_id=5LCW4SBCHQSVW4VIKAFSDDTXSFTXF5VMPA52C36XWMIMI5JXDEJQ, seg_pos={0 0}, upload=473668]: Long tail cancellation due to optimal threshold Dec 28 03:35:44.833468509 (540h6m44.704551544s) [noise=true , address=145.79.227.10:4402 , piece_id=GR34SYSSX2EMFHOX67HAOYQFHSE4QDUC2NSEDUHWYB6S2SZWQ7MA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 28 03:35:44.833477135 (540h6m44.70456018s) [noise=true , address=145.79.227.10:4402 , piece_id=GR34SYSSX2EMFHOX67HAOYQFHSE4QDUC2NSEDUHWYB6S2SZWQ7MA, seg_pos={0 0}, upload=473668]: Long tail cancellation due to optimal threshold Dec 28 03:35:44.833525548 (540h6m44.704608593s) [noise=true , address=145.79.227.5:4223 , piece_id=STQSOR4RKXJQIFZZQCSH3J4FPTS6CB4PNKGPRGTNM6I7BNSLHYRA, seg_pos={0 0}, upload=473668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:34460->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:34460->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34460->145.79.227.5:4223: use of closed network connection Dec 28 03:35:44.833555123 (540h6m44.704638158s) [noise=true , address=145.79.227.5:4223 , piece_id=STQSOR4RKXJQIFZZQCSH3J4FPTS6CB4PNKGPRGTNM6I7BNSLHYRA, seg_pos={0 0}, upload=473668]: Long tail cancellation due to optimal threshold Dec 28 03:35:44.833636338 (540h6m44.704719373s) [seg_pos={0 0}, upload=473668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:35:44.833751537 (540h6m44.704834572s) [upload=473668]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:35:44.833758581 (540h6m44.704841616s) [upload=473668]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:35:45.127555028 (540h6m44.998638063s) [upload=473668]: Done uploading segments... Dec 28 03:35:45.127573944 (540h6m44.99865698s) [upload=473668]: Upload finished. err: <nil> Duration: 30.003080357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........X....X...........X.....XX.....X..X.X....X.XXXX....X.] duration: 4.506550443s trace: https://jaeger.dev.storj.io/trace/5cfee049c00fc4e6 FAIL (2025-12-28 03:35:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.506578438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 19.61304164s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:35:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 19.613060575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 49.987612304s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:34:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 49.987630518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX.] duration: 6.028569024s trace: https://jaeger.dev.storj.io/trace/64419dc88841d4cb FAIL (2025-12-28 03:34:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ) context deadline exceeded ======LOGS====== Dec 28 03:33:45.747430799 (540h4m45.618513834s) [upload=473636]: Starting upload Dec 28 03:33:45.747491955 (540h4m45.618575s) [upload=473636]: Done starting upload Dec 28 03:33:45.747602135 (540h4m45.61868517s) [upload=473636]: Uploading segments... Dec 28 03:33:45.747766648 (540h4m45.618849683s) [upload=473636]: Got next segment. Inline: false Dec 28 03:33:45.747806584 (540h4m45.618889619s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:33:45.747810793 (540h4m45.618893838s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:33:45.997117805 (540h4m45.86820084s) [seg_pos={0 0}, upload=473636]: Begin upload segment... Dec 28 03:33:45.997173401 (540h4m45.868256436s) [seg_pos={0 0}, upload=473636]: Done begin upload segment. Dec 28 03:33:45.997214609 (540h4m45.868297644s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997299420 (540h4m45.868382455s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997729630 (540h4m45.868812665s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998073394 (540h4m45.869156429s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998402902 (540h4m45.869485937s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998942348 (540h4m45.870025393s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999343723 (540h4m45.870426758s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999755608 (540h4m45.870838653s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.000274987 (540h4m45.871358042s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.001711341 (540h4m45.872794407s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.003097301 (540h4m45.874180366s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.006293918 (540h4m45.877376953s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.007088891 (540h4m45.878171926s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.009599753 (540h4m45.880682788s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.013325828 (540h4m45.884408863s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.015511729 (540h4m45.886594774s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:47.734629898 (540h4m47.605712933s) [upload=473636]: Next returned nil segment Dec 28 03:33:47.734645909 (540h4m47.605728945s) [upload=473636]: Waiting for error group managing segments... Dec 28 03:33:48.020546311 (540h4m47.891629356s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035055670 (540h4m47.906138705s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035313530 (540h4m47.906396575s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.039157541 (540h4m47.910240576s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.089233874 (540h4m47.960316919s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.113787268 (540h4m47.984870303s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.287201077 (540h4m48.158284122s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.289709503 (540h4m48.160792538s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.347828651 (540h4m48.218911687s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.401863082 (540h4m48.272946127s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.720104827 (540h4m48.591187862s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.160352403 (540h4m49.031435438s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162353343 (540h4m49.033436389s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162365937 (540h4m49.033448982s) [seg_pos={0 0}, upload=473636]: Segment reached optimal threshold of 13 pieces. Dec 28 03:33:49.162672972 (540h4m49.033756007s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162682690 (540h4m49.033765735s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162781358 (540h4m49.033864413s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162790976 (540h4m49.033874011s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162857563 (540h4m49.033940598s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162864236 (540h4m49.033947271s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162882872 (540h4m49.033965917s) [seg_pos={0 0}, upload=473636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:33:49.162951192 (540h4m49.034034237s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:33:49.162957624 (540h4m49.034040669s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:33:49.519508356 (540h4m49.390591401s) [upload=473636]: Done uploading segments... Dec 28 03:33:49.519533103 (540h4m49.390616138s) [upload=473636]: Upload finished. err: <nil> Duration: 6.028596036s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X........X....X...........X.....XX.....X..X.X....X.XXXX....X] duration: 5.001557241s trace: https://jaeger.dev.storj.io/trace/5cfee049c00fc4e6 FAIL (2025-12-28 03:34:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001579525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXXX] duration: 30.002556657s trace: https://jaeger.dev.storj.io/trace/64419dc88841d4cb FAIL (2025-12-28 03:34:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ) context deadline exceeded ======LOGS====== Dec 28 03:33:45.747430799 (540h4m45.618513834s) [upload=473636]: Starting upload Dec 28 03:33:45.747491955 (540h4m45.618575s) [upload=473636]: Done starting upload Dec 28 03:33:45.747602135 (540h4m45.61868517s) [upload=473636]: Uploading segments... Dec 28 03:33:45.747766648 (540h4m45.618849683s) [upload=473636]: Got next segment. Inline: false Dec 28 03:33:45.747806584 (540h4m45.618889619s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:33:45.747810793 (540h4m45.618893838s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:33:45.997117805 (540h4m45.86820084s) [seg_pos={0 0}, upload=473636]: Begin upload segment... Dec 28 03:33:45.997173401 (540h4m45.868256436s) [seg_pos={0 0}, upload=473636]: Done begin upload segment. Dec 28 03:33:45.997214609 (540h4m45.868297644s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997299420 (540h4m45.868382455s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.997729630 (540h4m45.868812665s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998073394 (540h4m45.869156429s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998402902 (540h4m45.869485937s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.998942348 (540h4m45.870025393s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999343723 (540h4m45.870426758s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:45.999755608 (540h4m45.870838653s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.000274987 (540h4m45.871358042s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.001711341 (540h4m45.872794407s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.003097301 (540h4m45.874180366s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.006293918 (540h4m45.877376953s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.007088891 (540h4m45.878171926s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.009599753 (540h4m45.880682788s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.013325828 (540h4m45.884408863s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:46.015511729 (540h4m45.886594774s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Uploading piece... Dec 28 03:33:47.734629898 (540h4m47.605712933s) [upload=473636]: Next returned nil segment Dec 28 03:33:47.734645909 (540h4m47.605728945s) [upload=473636]: Waiting for error group managing segments... Dec 28 03:33:48.020546311 (540h4m47.891629356s) [noise=true , address=145.79.227.9:4365 , piece_id=XGQGQGBURJKBOV3M3SRHTVXWYO2YE7ABFPRXZ4D2I553L7PLG4WA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035055670 (540h4m47.906138705s) [noise=true , address=145.79.227.7:4293 , piece_id=DSRWWMRCSVIRM34AFY53F2QZ5RKWMSUNHG2GQQEOIQMO464THW6Q, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.035313530 (540h4m47.906396575s) [noise=true , address=145.79.227.48:4303 , piece_id=GN2XZTIUTIC4A5PZ6BLN2GGMBAOO6CJKJZN3NLGYD5YGRX43BPEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.039157541 (540h4m47.910240576s) [noise=true , address=145.79.227.51:4411 , piece_id=3S3X3QZJNXJEGP653NBUXXPNVLBI6PJGLGWLZDZXO4BAWCGM2BUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.089233874 (540h4m47.960316919s) [noise=true , address=145.79.227.46:4602 , piece_id=74K42IF2DNS2MNJKVIBZFGK6LVAQ2CQKB434RJQZ24MZYHSGWMOQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.113787268 (540h4m47.984870303s) [noise=true , address=145.79.227.6:4257 , piece_id=OHY5N2KNHJX6CHW2H4UNID5CMO5GJRVSWSKQFFW4TEA74A7NWTDA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.287201077 (540h4m48.158284122s) [noise=true , address=45.91.116.14:4191 , piece_id=HATL5PHE2QTM3PA4FX2NT3R4XAAJXBATQUAV3YHNZD36APEJVNWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.289709503 (540h4m48.160792538s) [noise=true , address=45.91.116.11:4117 , piece_id=7RK5KBXZMPYNX7N6NGW36XNT6JOSWATLH3ZKMQGBDBDLJU4J2GEA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.347828651 (540h4m48.218911687s) [noise=true , address=45.91.116.12:4226 , piece_id=4TLXZCH6M2XSQ6EWP3BITNBVGZQK4ZW2XY3PN5FLXH3US6QV5YWA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.401863082 (540h4m48.272946127s) [noise=true , address=45.91.116.9:4263 , piece_id=SDM4VLUYKHHCETUCAEROPN3HYG2M6BX2Z2S6PJI2WXGKOGEWQMWQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:48.720104827 (540h4m48.591187862s) [noise=true , address=45.91.116.13:4153 , piece_id=2R3K6S5IGPYWRH5AA7X4CHGBZLD4ALGYXK3MFERZ6EJ43CTLKTOA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.160352403 (540h4m49.031435438s) [noise=true , address=145.79.227.10:4402 , piece_id=MN44X67FPJUQGNDU4ZZGBH3XAPAEMKWZCDZ5X2ZRXRKMZ7GKMRCA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162353343 (540h4m49.033436389s) [noise=true , address=145.79.227.36:4518 , piece_id=6FRD3VQZOA3PTQDPWBCU6WJT4ZJXVBHHBC6BDEIYEYHKEQF777CQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: <nil> Dec 28 03:33:49.162365937 (540h4m49.033448982s) [seg_pos={0 0}, upload=473636]: Segment reached optimal threshold of 13 pieces. Dec 28 03:33:49.162672972 (540h4m49.033756007s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162682690 (540h4m49.033765735s) [noise=true , address=145.79.227.14:4840 , piece_id=ZK32CZLFR7B3IS5NR6GWDDS5IODVRV2IWSXBG6LOOZPGMKLNFJUA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162781358 (540h4m49.033864413s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162790976 (540h4m49.033874011s) [noise=true , address=77.90.31.15:4697 , piece_id=TGE77DHSPACJC2HVNDSEAZVAVLZHWF5T3XF3RXVU4FVRFEBJWCHA, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162857563 (540h4m49.033940598s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a9UG3xktEYLqGrk6Z1YRdzSd7EF8CpjJFLdLHi46PUXS92rSg): protocol: expected piece hash; context canceled; EOF Dec 28 03:33:49.162864236 (540h4m49.033947271s) [noise=true , address=145.79.227.27:4691 , piece_id=XBNAL44FQL6ERUA7BNL6FIZKM3O5W3ZOW3VPPGC6PEMA4PDO4PVQ, seg_pos={0 0}, upload=473636]: Long tail cancellation due to optimal threshold Dec 28 03:33:49.162882872 (540h4m49.033965917s) [seg_pos={0 0}, upload=473636]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:33:49.162951192 (540h4m49.034034237s) [upload=473636]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:33:49.162957624 (540h4m49.034040669s) [upload=473636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:33:49.519508356 (540h4m49.390591401s) [upload=473636]: Done uploading segments... Dec 28 03:33:49.519533103 (540h4m49.390616138s) [upload=473636]: Upload finished. err: <nil> Duration: 30.002579711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 56.309294596s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:34:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 56.309310476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 52 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXXX] duration: 30.003677008s trace: https://jaeger.dev.storj.io/trace/ad79903e61ccdc5 FAIL (2025-12-28 03:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 03:32:45.432920286 (540h3m45.304003321s) [upload=473623]: Starting upload Dec 28 03:32:45.432988536 (540h3m45.304071571s) [upload=473623]: Done starting upload Dec 28 03:32:45.433054260 (540h3m45.304137295s) [upload=473623]: Uploading segments... Dec 28 03:32:45.433159150 (540h3m45.304242195s) [upload=473623]: Got next segment. Inline: false Dec 28 03:32:45.433207733 (540h3m45.304290778s) [upload=473623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:32:45.433212031 (540h3m45.304295066s) [upload=473623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:32:45.684023348 (540h3m45.555106393s) [seg_pos={0 0}, upload=473623]: Begin upload segment... Dec 28 03:32:45.684083673 (540h3m45.555166728s) [seg_pos={0 0}, upload=473623]: Done begin upload segment. Dec 28 03:32:45.684127787 (540h3m45.555210832s) [noise=true , address=145.79.227.50:4373 , piece_id=TOCN7YWQIHPOJGSA3ZKWNHQXO5LW7OVSIQ3ZLELL5QJKFCR3YM3Q, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.684218569 (540h3m45.555301604s) [noise=true , address=145.79.227.39:4014 , piece_id=2WPHA5HX2RFVYW7DXADOHHI6IXNO3IQC4WTQI32BGJNM4PA6LJ6A, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.684836176 (540h3m45.555919221s) [noise=true , address=145.79.227.3:4150 , piece_id=EHNKMGGIXZ6R3MWXFJJMYGE7Z3IMTFUUTV7DB7URIITI3YSFN7GA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.685309237 (540h3m45.556392282s) [noise=true , address=145.79.227.48:4303 , piece_id=WCGQ3G6LPKRNVGLFZOI6LRAKU2UCP4W2RVTW6OERM4UUBF43SXYA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.685787818 (540h3m45.556870863s) [noise=true , address=45.91.116.10:4083 , piece_id=C4CXR3ESUIIAN4GN3KKGEAWVNLML6BF5N63ST7ODJJR2MC7V6I5A, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.686294613 (540h3m45.557377658s) [noise=true , address=77.90.31.13:4626 , piece_id=QNRDQ7XH7AOT5736RYPUE4UKTABGEQYWPXENXFIRVKLLGSFAGBVA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.686820805 (540h3m45.55790385s) [noise=true , address=145.79.227.13:4803 , piece_id=52UFG7YBZKWRUZAQZ67JIKJRI264RD4U2J4N7LMMW4HSNSUAGLGQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.687243279 (540h3m45.558326314s) [noise=true , address=145.79.227.41:4087 , piece_id=VUUUFYEJQQ3TAOS6TUSYNWM4F52ASJKW6Z6M2MU6DCSNFXKV2CRA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.687709066 (540h3m45.558792112s) [noise=true , address=45.91.116.12:4227 , piece_id=J6ZKJWDCY3VU6O3UKFQJ52ZBF3LKD46D3P3XYRKJYC224D5D43AQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.689573326 (540h3m45.560656371s) [noise=true , address=145.79.227.34:4907 , piece_id=OQBOWOUHQJTE7RAPL5AYLHJQS5QFSZ6YHF5N2R6SKTV7SPREOZQQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.689987805 (540h3m45.56107084s) [noise=true , address=77.90.31.12:4589 , piece_id=UQKQEFGS5RXNWBGJZIVJBKYCBK53Q3FGAFN4KIAO3S3R2OIAWDXA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.691992142 (540h3m45.563075187s) [noise=true , address=77.90.31.19:4841 , piece_id=QWRQFEG2QCIUSCJR2BADBU45NS2CZQ55EYRAX2QFJWUF3NHGC4HQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.693969768 (540h3m45.565052813s) [noise=true , address=145.79.227.36:4518 , piece_id=CBYXXQCS6HKYC2RHHZEJDYGJIL5KBI76DGI4VNMLB2LEHS4V75QA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.696001777 (540h3m45.567084832s) [noise=true , address=145.79.227.12:4767 , piece_id=SLQPZHCS5FS4FXAM6N6INOZKOTZYD3F5EJ62PFZ6NR3CHAGORFRA, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.697787257 (540h3m45.568870302s) [noise=true , address=45.91.116.8:4045 , piece_id=75B5J6C6NUZ55MWQDM3PTPUHFCLSTVMHFUS3FJWC2UWG6K3HACVQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:45.699666956 (540h3m45.570750001s) [noise=true , address=45.91.116.9:4263 , piece_id=W2OP4HIHE7LJ7KMKYA5SJU544WWKD6CYZ5L5XOC3CJ64ZQLSF6ZQ, seg_pos={0 0}, upload=473623]: Uploading piece... Dec 28 03:32:47.470574951 (540h3m47.341657996s) [upload=473623]: Next returned nil segment Dec 28 03:32:47.470598706 (540h3m47.341681751s) [upload=473623]: Waiting for error group managing segments... Dec 28 03:32:47.767904355 (540h3m47.6389874s) [noise=true , address=145.79.227.41:4087 , piece_id=VUUUFYEJQQ3TAOS6TUSYNWM4F52ASJKW6Z6M2MU6DCSNFXKV2CRA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.775433041 (540h3m47.646516076s) [noise=true , address=145.79.227.3:4150 , piece_id=EHNKMGGIXZ6R3MWXFJJMYGE7Z3IMTFUUTV7DB7URIITI3YSFN7GA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.791343157 (540h3m47.662426222s) [noise=true , address=145.79.227.50:4373 , piece_id=TOCN7YWQIHPOJGSA3ZKWNHQXO5LW7OVSIQ3ZLELL5QJKFCR3YM3Q, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.903554436 (540h3m47.774637481s) [noise=true , address=77.90.31.19:4841 , piece_id=QWRQFEG2QCIUSCJR2BADBU45NS2CZQ55EYRAX2QFJWUF3NHGC4HQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:47.965550146 (540h3m47.836633191s) [noise=true , address=77.90.31.12:4589 , piece_id=UQKQEFGS5RXNWBGJZIVJBKYCBK53Q3FGAFN4KIAO3S3R2OIAWDXA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.034511110 (540h3m47.905594155s) [noise=true , address=45.91.116.8:4045 , piece_id=75B5J6C6NUZ55MWQDM3PTPUHFCLSTVMHFUS3FJWC2UWG6K3HACVQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.061535999 (540h3m47.932619034s) [noise=true , address=45.91.116.9:4263 , piece_id=W2OP4HIHE7LJ7KMKYA5SJU544WWKD6CYZ5L5XOC3CJ64ZQLSF6ZQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.285918931 (540h3m48.157001967s) [noise=true , address=145.79.227.48:4303 , piece_id=WCGQ3G6LPKRNVGLFZOI6LRAKU2UCP4W2RVTW6OERM4UUBF43SXYA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.321590296 (540h3m48.192673331s) [noise=true , address=145.79.227.13:4803 , piece_id=52UFG7YBZKWRUZAQZ67JIKJRI264RD4U2J4N7LMMW4HSNSUAGLGQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.324041594 (540h3m48.19512464s) [noise=true , address=145.79.227.36:4518 , piece_id=CBYXXQCS6HKYC2RHHZEJDYGJIL5KBI76DGI4VNMLB2LEHS4V75QA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.354092186 (540h3m48.225175221s) [noise=true , address=45.91.116.12:4227 , piece_id=J6ZKJWDCY3VU6O3UKFQJ52ZBF3LKD46D3P3XYRKJYC224D5D43AQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.560567850 (540h3m48.431650895s) [noise=true , address=77.90.31.13:4626 , piece_id=QNRDQ7XH7AOT5736RYPUE4UKTABGEQYWPXENXFIRVKLLGSFAGBVA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.861348917 (540h3m48.732431952s) [noise=true , address=145.79.227.39:4014 , piece_id=2WPHA5HX2RFVYW7DXADOHHI6IXNO3IQC4WTQI32BGJNM4PA6LJ6A, seg_pos={0 0}, upload=473623]: Done uploading piece. err: <nil> Dec 28 03:32:48.861369105 (540h3m48.73245214s) [seg_pos={0 0}, upload=473623]: Segment reached optimal threshold of 13 pieces. Dec 28 03:32:48.861606227 (540h3m48.732689262s) [noise=true , address=145.79.227.34:4907 , piece_id=OQBOWOUHQJTE7RAPL5AYLHJQS5QFSZ6YHF5N2R6SKTV7SPREOZQQ, seg_pos={0 0}, upload=473623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:48.861613130 (540h3m48.732696165s) [noise=true , address=145.79.227.34:4907 , piece_id=OQBOWOUHQJTE7RAPL5AYLHJQS5QFSZ6YHF5N2R6SKTV7SPREOZQQ, seg_pos={0 0}, upload=473623]: Long tail cancellation due to optimal threshold Dec 28 03:32:48.861987543 (540h3m48.733070588s) [noise=true , address=45.91.116.10:4083 , piece_id=C4CXR3ESUIIAN4GN3KKGEAWVNLML6BF5N63ST7ODJJR2MC7V6I5A, seg_pos={0 0}, upload=473623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:41156->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 03:32:48.862002913 (540h3m48.733085958s) [noise=true , address=45.91.116.10:4083 , piece_id=C4CXR3ESUIIAN4GN3KKGEAWVNLML6BF5N63ST7ODJJR2MC7V6I5A, seg_pos={0 0}, upload=473623]: Long tail cancellation due to optimal threshold Dec 28 03:32:48.862056544 (540h3m48.733139599s) [noise=true , address=145.79.227.12:4767 , piece_id=SLQPZHCS5FS4FXAM6N6INOZKOTZYD3F5EJ62PFZ6NR3CHAGORFRA, seg_pos={0 0}, upload=473623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:48.862066213 (540h3m48.733149248s) [noise=true , address=145.79.227.12:4767 , piece_id=SLQPZHCS5FS4FXAM6N6INOZKOTZYD3F5EJ62PFZ6NR3CHAGORFRA, seg_pos={0 0}, upload=473623]: Long tail cancellation due to optimal threshold Dec 28 03:32:48.862086552 (540h3m48.733169587s) [seg_pos={0 0}, upload=473623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:32:48.862156365 (540h3m48.73323941s) [upload=473623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:32:48.862161033 (540h3m48.733244068s) [upload=473623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:32:49.172333874 (540h3m49.043416919s) [upload=473623]: Done uploading segments... Dec 28 03:32:49.172354133 (540h3m49.043437168s) [upload=473623]: Upload finished. err: <nil> Duration: 30.003702196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 24.330730747s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:33:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 24.330748532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.X........X....X...........X.....XX.....X..X.X....X.XXXX....] duration: 4.864477707s trace: https://jaeger.dev.storj.io/trace/58e23bd7ac68184a FAIL (2025-12-28 03:33:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.86450639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 53.289843292s trace: https://jaeger.dev.storj.io/trace/54422f75ff432c0 FAIL (2025-12-28 03:32:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720215866 (539h54m35.591298911s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720567665 (539h54m35.5916507s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.720883607 (539h54m35.591966652s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.721874904 (539h54m35.592957949s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722249658 (539h54m35.593332693s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.722786881 (539h54m35.593869926s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723112100 (539h54m35.594195135s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723440756 (539h54m35.594523791s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.723849544 (539h54m35.594932589s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724241290 (539h54m35.595324335s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724607808 (539h54m35.595690854s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.724974035 (539h54m35.59605707s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.725302281 (539h54m35.596385316s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:35.726824420 (539h54m35.597907455s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Uploading piece... Dec 28 03:23:37.063163473 (539h54m36.934246518s) [upload=473490]: Next returned nil segment Dec 28 03:23:37.063176499 (539h54m36.934259534s) [upload=473490]: Waiting for error group managing segments... Dec 28 03:23:37.416516311 (539h54m37.287599376s) [noise=true , address=64.49.13.140:4513 , piece_id=Y3HW5I2FX2CWSPVUS7CU3JPNN2RHP2ASL757AUYINBKRKCLXAJGQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.606629586 (539h54m37.477712622s) [noise=true , address=145.79.227.27:4723 , piece_id=4DECSGDNZLXNQVPU7QALLPFQYDO7MOG6TIBNWQ5UHNVCIHB4GUYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.617730204 (539h54m37.488813249s) [noise=true , address=145.79.227.26:4669 , piece_id=5D7SGVO274FEZ4JVESTMUYQ3RZ3DJA5DL2QZV5YKMXEB4TM6PHAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.632542950 (539h54m37.503625995s) [noise=true , address=145.79.227.38:4989 , piece_id=OUINZ6BXOLFCDNCXMX7PPFSKWMJ7IPZWO7DAGLGVV2AB7UHO3JYQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.643987531 (539h54m37.515070566s) [noise=true , address=145.79.227.47:4272 , piece_id=FL2U4W2YCV5ANDVFHB7H4TFT3SYTBGNLI3KZDI7MY2K6NTSMRUAA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.652913619 (539h54m37.523996654s) [noise=true , address=145.79.227.18:4040 , piece_id=W55GK3QDHRHNR6YJ7JKVZMIEKFQLZSTXKSQ5OMW3U7XRI44EM6RA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.661966357 (539h54m37.533049392s) [noise=true , address=145.79.227.37:4569 , piece_id=DUXSQUT36XUF66ZVQ2V7ZWNXA6JZKBDO5RXT3QVT26GE2FTH3QXQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.778627848 (539h54m37.649710893s) [noise=true , address=77.90.31.11:4561 , piece_id=HIUCZRRZOMU4RBYZU5SADJAOS4RVML2I6PILIX2V65RS22EBN42Q, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.815974812 (539h54m37.687057857s) [noise=true , address=45.91.116.14:4217 , piece_id=SPQSUVHIHI32ZMIRE4NHSNJCMWS4NS6ZZPYF6FV7FQH6VADQIV5A, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:37.861731780 (539h54m37.732814815s) [noise=true , address=77.90.31.14:4673 , piece_id=ZIB5VOFL7PJSRFDV3HP5XAKYHOJXRVHCYDPIHKRYPNM5ROAY5DTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.093273211 (539h54m37.964356256s) [noise=true , address=77.90.31.11:4587 , piece_id=6467VZCLWHJTZ36ENMEOEOU2E4MAUCDL2XHNAOMJTCCTUSN33NFA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.267694395 (539h54m38.13877745s) [noise=true , address=77.90.31.17:4779 , piece_id=TLU5ZLZRVAMGQLSE2LLNIUE4WOWJBMFJGFQKC3KOUJ23V3UVFMRA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.540947761 (539h54m38.412030796s) [noise=true , address=145.79.227.40:4062 , piece_id=RCVKQLDSWIWX2FQBTUFPEUXQZLP2C3ZCLV2JWHOV7CMJ7ZBNWZNQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815393527 (539h54m38.686476562s) [noise=true , address=145.79.227.43:4187 , piece_id=MRILXYFSWVSH35ZCCFZWLEXENWJII6G6LOR2N277GIBMI4PXPPTQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: <nil> Dec 28 03:23:38.815412894 (539h54m38.686495939s) [seg_pos={0 0}, upload=473490]: Segment reached optimal threshold of 14 pieces. Dec 28 03:23:38.815594951 (539h54m38.686677986s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43502->64.49.13.169:4715: use of closed network connection Dec 28 03:23:38.815601032 (539h54m38.686684077s) [noise=true , address=64.49.13.169:4715 , piece_id=FRVNWFP2AE4DXE4RKZRE6WTK43T74BTTYM2U7E4P3B4XZIMC2QMQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815826983 (539h54m38.686910018s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815834507 (539h54m38.686917542s) [noise=true , address=145.79.227.41:4118 , piece_id=C7JJ3NHR5355M64G2EF2WWERQVGWR243RQHDV3GBAIC3A66JDMHQ, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.815890433 (539h54m38.686973468s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o): protocol: expected piece hash; context canceled; EOF Dec 28 03:23:38.815914619 (539h54m38.686997654s) [noise=true , address=145.79.227.33:4896 , piece_id=7NRM7I4IDHW2WGOCG3B6L6HN74ANXWDJCFRPBM4BCHWJELSNNUXA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816105613 (539h54m38.687188648s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NQzTwVYQa6ZzxAHQm1apkwWnqJstTnAJQyHKhBzC2jTN9kEVK): protocol: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58940->145.79.227.28:4731: use of closed network connection Dec 28 03:23:38.816112426 (539h54m38.687195461s) [noise=true , address=145.79.227.28:4731 , piece_id=EPW3LUA3EWRCURHZ66NCRCA36CNIW4GYQMFNKPZZTRCB3YBGTKJA, seg_pos={0 0}, upload=473490]: Long tail cancellation due to optimal threshold Dec 28 03:23:38.816128436 (539h54m38.687211481s) [seg_pos={0 0}, upload=473490]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 03:23:38.816257582 (539h54m38.687340627s) [upload=473490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:23:38.816265778 (539h54m38.687348813s) [upload=473490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:23:39.111115407 (539h54m38.982198442s) [upload=473490]: Done uploading segments... Dec 28 03:23:39.111138972 (539h54m38.982222008s) [upload=473490]: Upload finished. err: <nil> Duration: 53.289862168s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 51 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXX.XXXXXXXXX.X..XX..XXXXXX.XXXXXXXXXXXXXXXXX.XXX] duration: 30.004599396s trace: https://jaeger.dev.storj.io/trace/31ec26dc84c3e512 FAIL (2025-12-28 03:32:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 7M5CYZHNS7LFSHJ7FPXXNZBTEWRRCIWK2ZMDFKZLPGJRFKLRE4YA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: Q4RKNSG3EFMPT2ERPWEOAISOKQ2IKDQQHYHAHZMHTYCHP5MI4QRQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: QMVRPGAB7U7AE5N3UABMOBK2BJJSAIYAPYVSLJLKKASKIXXJJSRQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 27AVXPXTMI7JL242QAZJ7FNCBH3J5HNGPHNAJ23HD5ZUXJUFLBQA) context deadline exceeded ======LOGS====== Dec 28 03:32:13.492523947 (540h3m13.363606982s) [upload=473617]: Starting upload Dec 28 03:32:13.492584753 (540h3m13.363667798s) [upload=473617]: Done starting upload Dec 28 03:32:13.492681097 (540h3m13.363764132s) [upload=473617]: Uploading segments... Dec 28 03:32:13.492784273 (540h3m13.363867309s) [upload=473617]: Got next segment. Inline: false Dec 28 03:32:13.492821013 (540h3m13.363904048s) [upload=473617]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 03:32:13.492827175 (540h3m13.36391023s) [upload=473617]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 03:32:13.737278110 (540h3m13.608361146s) [seg_pos={0 0}, upload=473617]: Begin upload segment... Dec 28 03:32:13.737331211 (540h3m13.608414246s) [seg_pos={0 0}, upload=473617]: Done begin upload segment. Dec 28 03:32:13.737423387 (540h3m13.608506432s) [noise=true , address=45.91.116.12:4226 , piece_id=A2XYJO3YSMGLEETT3B2EFATUSAQYETIU677ILBXVF3CQM7HMRITQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.737439878 (540h3m13.608522913s) [noise=true , address=77.90.31.19:4844 , piece_id=OGNDXD2GDQ4D2MTXWKLFNSSS6D2SJWQIKB4MTI2NF47RYB33MKOQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.738017999 (540h3m13.609101044s) [noise=true , address=145.79.227.13:4803 , piece_id=TGFZYOG6U46MLIJW5KNKVXA7RRETQQHQKNBVGUCL4EYAI4DHG3KQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.738369869 (540h3m13.609452904s) [noise=true , address=45.91.116.7:4012 , piece_id=QMVRPGAB7U7AE5N3UABMOBK2BJJSAIYAPYVSLJLKKASKIXXJJSRQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.738758579 (540h3m13.609841624s) [noise=true , address=145.79.227.52:4445 , piece_id=N3EQPQA4MNUNV343NBC326N3QNNAHOIO57BGLEXMFNYRJC7UAYEA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.739114177 (540h3m13.610197212s) [noise=true , address=45.91.116.14:4192 , piece_id=27AVXPXTMI7JL242QAZJ7FNCBH3J5HNGPHNAJ23HD5ZUXJUFLBQA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.739449465 (540h3m13.6105325s) [noise=true , address=145.79.227.17:4971 , piece_id=NIRGOZ4GZMFAO4RYI2LMU5LTI3PAZZR6OP7W7IWDINC6VZRXYFFQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.739861930 (540h3m13.610944975s) [noise=true , address=45.91.116.10:4082 , piece_id=EKPQZ3FD6HTXWK4WPCZ5RQLPCHLHPA7TJHCPQ4HE3XGVJIF3MMWQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.740696800 (540h3m13.611779835s) [noise=true , address=145.79.227.11:4732 , piece_id=CLTFD7G455ITSS3H5UE4W5C4JFE6SMARMUHNIW4VHU5IPCMNW2VA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.741769413 (540h3m13.612852458s) [noise=true , address=77.90.31.16:4735 , piece_id=7M5CYZHNS7LFSHJ7FPXXNZBTEWRRCIWK2ZMDFKZLPGJRFKLRE4YA, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.742052101 (540h3m13.613135136s) [noise=true , address=145.79.227.15:4875 , piece_id=CXODGGZMXY6YBWX2R2KIP46WXICJO7VFWS3WOGPNRDBJS3Q4OKSQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.743410868 (540h3m13.614493903s) [noise=true , address=45.91.116.5:4297 , piece_id=Q4RKNSG3EFMPT2ERPWEOAISOKQ2IKDQQHYHAHZMHTYCHP5MI4QRQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.744673151 (540h3m13.615756187s) [noise=true , address=145.79.227.27:4691 , piece_id=QSZTQRPW4OEDJ7KBS3F5REBBGCHNXW3CKBCKLU635B3X45VOY2EQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.745903385 (540h3m13.61698643s) [noise=true , address=145.79.227.45:4229 , piece_id=UNJV7CQVHNDHXUZJDOE3GQ44EMXV425BYDYJYRFNM2IXYTW7E45A, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.746211340 (540h3m13.617294385s) [noise=true , address=77.90.31.13:4627 , piece_id=KTVTTKB7CLV266YSAVHD32GK2QM4PEFDWYLII5ZH5XJA4KKQXVVQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:13.750079256 (540h3m13.621162291s) [noise=true , address=145.79.227.50:4373 , piece_id=ZD4BLOXH7CZJRA2VBKS7HPEQDPMM4SD2G6UF24W25SEAJKXMI7ZQ, seg_pos={0 0}, upload=473617]: Uploading piece... Dec 28 03:32:15.464465544 (540h3m15.335548579s) [upload=473617]: Next returned nil segment Dec 28 03:32:15.464493097 (540h3m15.335576142s) [upload=473617]: Waiting for error group managing segments... Dec 28 03:32:15.758258825 (540h3m15.62934187s) [noise=true , address=145.79.227.52:4445 , piece_id=N3EQPQA4MNUNV343NBC326N3QNNAHOIO57BGLEXMFNYRJC7UAYEA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.759138911 (540h3m15.630221946s) [noise=true , address=145.79.227.50:4373 , piece_id=ZD4BLOXH7CZJRA2VBKS7HPEQDPMM4SD2G6UF24W25SEAJKXMI7ZQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.766146685 (540h3m15.63722972s) [noise=true , address=145.79.227.13:4803 , piece_id=TGFZYOG6U46MLIJW5KNKVXA7RRETQQHQKNBVGUCL4EYAI4DHG3KQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.773461294 (540h3m15.644544339s) [noise=true , address=145.79.227.27:4691 , piece_id=QSZTQRPW4OEDJ7KBS3F5REBBGCHNXW3CKBCKLU635B3X45VOY2EQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.773573918 (540h3m15.644656953s) [noise=true , address=145.79.227.17:4971 , piece_id=NIRGOZ4GZMFAO4RYI2LMU5LTI3PAZZR6OP7W7IWDINC6VZRXYFFQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.918769555 (540h3m15.7898526s) [noise=true , address=77.90.31.16:4735 , piece_id=7M5CYZHNS7LFSHJ7FPXXNZBTEWRRCIWK2ZMDFKZLPGJRFKLRE4YA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.920111019 (540h3m15.791194054s) [noise=true , address=77.90.31.13:4627 , piece_id=KTVTTKB7CLV266YSAVHD32GK2QM4PEFDWYLII5ZH5XJA4KKQXVVQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:15.957983063 (540h3m15.829066098s) [noise=true , address=45.91.116.5:4297 , piece_id=Q4RKNSG3EFMPT2ERPWEOAISOKQ2IKDQQHYHAHZMHTYCHP5MI4QRQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.004341438 (540h3m15.875424483s) [noise=true , address=45.91.116.14:4192 , piece_id=27AVXPXTMI7JL242QAZJ7FNCBH3J5HNGPHNAJ23HD5ZUXJUFLBQA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.029985688 (540h3m15.901068733s) [noise=true , address=77.90.31.19:4844 , piece_id=OGNDXD2GDQ4D2MTXWKLFNSSS6D2SJWQIKB4MTI2NF47RYB33MKOQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.062195312 (540h3m15.933278347s) [noise=true , address=45.91.116.7:4012 , piece_id=QMVRPGAB7U7AE5N3UABMOBK2BJJSAIYAPYVSLJLKKASKIXXJJSRQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.091152611 (540h3m15.962235656s) [noise=true , address=45.91.116.12:4226 , piece_id=A2XYJO3YSMGLEETT3B2EFATUSAQYETIU677ILBXVF3CQM7HMRITQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.122380804 (540h3m15.99346384s) [noise=true , address=45.91.116.10:4082 , piece_id=EKPQZ3FD6HTXWK4WPCZ5RQLPCHLHPA7TJHCPQ4HE3XGVJIF3MMWQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: <nil> Dec 28 03:32:16.122399991 (540h3m15.993483027s) [seg_pos={0 0}, upload=473617]: Segment reached optimal threshold of 13 pieces. Dec 28 03:32:16.122643836 (540h3m15.993726871s) [noise=true , address=145.79.227.11:4732 , piece_id=CLTFD7G455ITSS3H5UE4W5C4JFE6SMARMUHNIW4VHU5IPCMNW2VA, seg_pos={0 0}, upload=473617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 188.166.225.170:36216->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:36216->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36216->145.79.227.11:4732: use of closed network connection Dec 28 03:32:16.122650258 (540h3m15.993733303s) [noise=true , address=145.79.227.11:4732 , piece_id=CLTFD7G455ITSS3H5UE4W5C4JFE6SMARMUHNIW4VHU5IPCMNW2VA, seg_pos={0 0}, upload=473617]: Long tail cancellation due to optimal threshold Dec 28 03:32:16.122942955 (540h3m15.99402599s) [noise=true , address=145.79.227.45:4229 , piece_id=UNJV7CQVHNDHXUZJDOE3GQ44EMXV425BYDYJYRFNM2IXYTW7E45A, seg_pos={0 0}, upload=473617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:16.122949969 (540h3m15.994033004s) [noise=true , address=145.79.227.45:4229 , piece_id=UNJV7CQVHNDHXUZJDOE3GQ44EMXV425BYDYJYRFNM2IXYTW7E45A, seg_pos={0 0}, upload=473617]: Long tail cancellation due to optimal threshold Dec 28 03:32:16.123062663 (540h3m15.994145708s) [noise=true , address=145.79.227.15:4875 , piece_id=CXODGGZMXY6YBWX2R2KIP46WXICJO7VFWS3WOGPNRDBJS3Q4OKSQ, seg_pos={0 0}, upload=473617]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 28 03:32:16.123068584 (540h3m15.994151619s) [noise=true , address=145.79.227.15:4875 , piece_id=CXODGGZMXY6YBWX2R2KIP46WXICJO7VFWS3WOGPNRDBJS3Q4OKSQ, seg_pos={0 0}, upload=473617]: Long tail cancellation due to optimal threshold Dec 28 03:32:16.123086187 (540h3m15.994169232s) [seg_pos={0 0}, upload=473617]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 03:32:16.123154668 (540h3m15.994237703s) [upload=473617]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 03:32:16.123159989 (540h3m15.994243024s) [upload=473617]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 03:32:16.410762552 (540h3m16.281845607s) [upload=473617]: Done uploading segments... Dec 28 03:32:16.410786316 (540h3m16.281869361s) [upload=473617]: Upload finished. err: <nil> Duration: 30.004621861s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)