Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........X.......................................] duration: 33.529162178s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:10:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 33.529193248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X......X..X...X..XX.X...XX.........X.................X.....] duration: 6.069802911s trace: https://jaeger.dev.storj.io/trace/b82db2838f61e63 FAIL (2025-12-26 22:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 22:05:24.864904308 (510h36m24.735987353s) [upload=448065]: Got next segment. Inline: false Dec 26 22:05:24.864952599 (510h36m24.736035654s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:05:24.864959422 (510h36m24.736042467s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:05:25.111861816 (510h36m24.982944861s) [seg_pos={0 0}, upload=448065]: Begin upload segment... Dec 26 22:05:25.111935336 (510h36m24.983018381s) [seg_pos={0 0}, upload=448065]: Done begin upload segment. Dec 26 22:05:25.111966516 (510h36m24.983049551s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.111991903 (510h36m24.983074938s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.112535639 (510h36m24.983618684s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113043667 (510h36m24.984126702s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113582021 (510h36m24.984665066s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114083656 (510h36m24.985166691s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114396211 (510h36m24.985479246s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114719127 (510h36m24.985802182s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115110252 (510h36m24.986193287s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115356320 (510h36m24.986439365s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116335635 (510h36m24.98741868s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116610929 (510h36m24.987693974s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116696833 (510h36m24.987779898s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117284651 (510h36m24.988367697s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117487418 (510h36m24.988570463s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117843225 (510h36m24.98892626s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:26.829253118 (510h36m26.700336153s) [upload=448065]: Next returned nil segment Dec 26 22:05:26.829266823 (510h36m26.700349868s) [upload=448065]: Waiting for error group managing segments... Dec 26 22:05:27.114702552 (510h36m26.985785587s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.117900832 (510h36m26.988983867s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.135399732 (510h36m27.006482767s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137100431 (510h36m27.008183476s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137226731 (510h36m27.008309776s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.143949792 (510h36m27.015032827s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.155016275 (510h36m27.02609932s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.168724899 (510h36m27.039807934s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.298348504 (510h36m27.169431549s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.338925720 (510h36m27.210008765s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.346580496 (510h36m27.217663531s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.426919254 (510h36m27.298002289s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460645785 (510h36m27.331728841s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460670171 (510h36m27.331753206s) [seg_pos={0 0}, upload=448065]: Segment reached optimal threshold of 13 pieces. Dec 26 22:05:27.460905920 (510h36m27.331988965s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection Dec 26 22:05:27.460913555 (510h36m27.3319966s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461068510 (510h36m27.332151565s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 22:05:27.461079521 (510h36m27.332162576s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461130388 (510h36m27.332213433s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection Dec 26 22:05:27.461138514 (510h36m27.332221549s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461255155 (510h36m27.3323382s) [seg_pos={0 0}, upload=448065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:05:27.461370555 (510h36m27.33245359s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:05:27.461377589 (510h36m27.332460624s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:05:27.758634566 (510h36m27.629717601s) [upload=448065]: Done uploading segments... Dec 26 22:05:27.758653101 (510h36m27.629736136s) [upload=448065]: Upload finished. err: <nil> Duration: 6.069832247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........X......................................] duration: 8.193733017s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:08:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 8.193759237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........X.....................................] duration: 6.916591845s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:08:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 6.91662051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X......X..X...X..XX.X...XX.........X.................X....] duration: 8.610419516s trace: https://jaeger.dev.storj.io/trace/b82db2838f61e63 FAIL (2025-12-26 22:07:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 22:05:24.864904308 (510h36m24.735987353s) [upload=448065]: Got next segment. Inline: false Dec 26 22:05:24.864952599 (510h36m24.736035654s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:05:24.864959422 (510h36m24.736042467s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:05:25.111861816 (510h36m24.982944861s) [seg_pos={0 0}, upload=448065]: Begin upload segment... Dec 26 22:05:25.111935336 (510h36m24.983018381s) [seg_pos={0 0}, upload=448065]: Done begin upload segment. Dec 26 22:05:25.111966516 (510h36m24.983049551s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.111991903 (510h36m24.983074938s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.112535639 (510h36m24.983618684s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113043667 (510h36m24.984126702s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113582021 (510h36m24.984665066s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114083656 (510h36m24.985166691s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114396211 (510h36m24.985479246s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114719127 (510h36m24.985802182s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115110252 (510h36m24.986193287s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115356320 (510h36m24.986439365s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116335635 (510h36m24.98741868s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116610929 (510h36m24.987693974s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116696833 (510h36m24.987779898s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117284651 (510h36m24.988367697s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117487418 (510h36m24.988570463s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117843225 (510h36m24.98892626s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:26.829253118 (510h36m26.700336153s) [upload=448065]: Next returned nil segment Dec 26 22:05:26.829266823 (510h36m26.700349868s) [upload=448065]: Waiting for error group managing segments... Dec 26 22:05:27.114702552 (510h36m26.985785587s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.117900832 (510h36m26.988983867s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.135399732 (510h36m27.006482767s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137100431 (510h36m27.008183476s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137226731 (510h36m27.008309776s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.143949792 (510h36m27.015032827s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.155016275 (510h36m27.02609932s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.168724899 (510h36m27.039807934s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.298348504 (510h36m27.169431549s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.338925720 (510h36m27.210008765s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.346580496 (510h36m27.217663531s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.426919254 (510h36m27.298002289s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460645785 (510h36m27.331728841s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460670171 (510h36m27.331753206s) [seg_pos={0 0}, upload=448065]: Segment reached optimal threshold of 13 pieces. Dec 26 22:05:27.460905920 (510h36m27.331988965s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection Dec 26 22:05:27.460913555 (510h36m27.3319966s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461068510 (510h36m27.332151565s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 22:05:27.461079521 (510h36m27.332162576s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461130388 (510h36m27.332213433s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection Dec 26 22:05:27.461138514 (510h36m27.332221549s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461255155 (510h36m27.3323382s) [seg_pos={0 0}, upload=448065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:05:27.461370555 (510h36m27.33245359s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:05:27.461377589 (510h36m27.332460624s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:05:27.758634566 (510h36m27.629717601s) [upload=448065]: Done uploading segments... Dec 26 22:05:27.758653101 (510h36m27.629736136s) [upload=448065]: Upload finished. err: <nil> Duration: 8.610440927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XX.X......X..X...X..XX.X...XX.........X.................X...] duration: 6.186773763s trace: https://jaeger.dev.storj.io/trace/b82db2838f61e63 FAIL (2025-12-26 22:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 22:05:24.864904308 (510h36m24.735987353s) [upload=448065]: Got next segment. Inline: false Dec 26 22:05:24.864952599 (510h36m24.736035654s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:05:24.864959422 (510h36m24.736042467s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:05:25.111861816 (510h36m24.982944861s) [seg_pos={0 0}, upload=448065]: Begin upload segment... Dec 26 22:05:25.111935336 (510h36m24.983018381s) [seg_pos={0 0}, upload=448065]: Done begin upload segment. Dec 26 22:05:25.111966516 (510h36m24.983049551s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.111991903 (510h36m24.983074938s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.112535639 (510h36m24.983618684s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113043667 (510h36m24.984126702s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113582021 (510h36m24.984665066s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114083656 (510h36m24.985166691s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114396211 (510h36m24.985479246s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114719127 (510h36m24.985802182s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115110252 (510h36m24.986193287s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115356320 (510h36m24.986439365s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116335635 (510h36m24.98741868s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116610929 (510h36m24.987693974s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116696833 (510h36m24.987779898s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117284651 (510h36m24.988367697s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117487418 (510h36m24.988570463s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117843225 (510h36m24.98892626s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:26.829253118 (510h36m26.700336153s) [upload=448065]: Next returned nil segment Dec 26 22:05:26.829266823 (510h36m26.700349868s) [upload=448065]: Waiting for error group managing segments... Dec 26 22:05:27.114702552 (510h36m26.985785587s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.117900832 (510h36m26.988983867s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.135399732 (510h36m27.006482767s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137100431 (510h36m27.008183476s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137226731 (510h36m27.008309776s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.143949792 (510h36m27.015032827s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.155016275 (510h36m27.02609932s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.168724899 (510h36m27.039807934s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.298348504 (510h36m27.169431549s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.338925720 (510h36m27.210008765s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.346580496 (510h36m27.217663531s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.426919254 (510h36m27.298002289s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460645785 (510h36m27.331728841s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460670171 (510h36m27.331753206s) [seg_pos={0 0}, upload=448065]: Segment reached optimal threshold of 13 pieces. Dec 26 22:05:27.460905920 (510h36m27.331988965s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection Dec 26 22:05:27.460913555 (510h36m27.3319966s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461068510 (510h36m27.332151565s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 22:05:27.461079521 (510h36m27.332162576s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461130388 (510h36m27.332213433s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection Dec 26 22:05:27.461138514 (510h36m27.332221549s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461255155 (510h36m27.3323382s) [seg_pos={0 0}, upload=448065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:05:27.461370555 (510h36m27.33245359s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:05:27.461377589 (510h36m27.332460624s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:05:27.758634566 (510h36m27.629717601s) [upload=448065]: Done uploading segments... Dec 26 22:05:27.758653101 (510h36m27.629736136s) [upload=448065]: Upload finished. err: <nil> Duration: 6.186794513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........X....................................] duration: 5.91750281s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:07:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 5.917528439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX.X......X..X...X..XX.X...XX.........X.................X..] duration: 6.236862051s trace: https://jaeger.dev.storj.io/trace/b82db2838f61e63 FAIL (2025-12-26 22:06:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 22:05:24.864904308 (510h36m24.735987353s) [upload=448065]: Got next segment. Inline: false Dec 26 22:05:24.864952599 (510h36m24.736035654s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:05:24.864959422 (510h36m24.736042467s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:05:25.111861816 (510h36m24.982944861s) [seg_pos={0 0}, upload=448065]: Begin upload segment... Dec 26 22:05:25.111935336 (510h36m24.983018381s) [seg_pos={0 0}, upload=448065]: Done begin upload segment. Dec 26 22:05:25.111966516 (510h36m24.983049551s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.111991903 (510h36m24.983074938s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.112535639 (510h36m24.983618684s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113043667 (510h36m24.984126702s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113582021 (510h36m24.984665066s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114083656 (510h36m24.985166691s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114396211 (510h36m24.985479246s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114719127 (510h36m24.985802182s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115110252 (510h36m24.986193287s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115356320 (510h36m24.986439365s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116335635 (510h36m24.98741868s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116610929 (510h36m24.987693974s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116696833 (510h36m24.987779898s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117284651 (510h36m24.988367697s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117487418 (510h36m24.988570463s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117843225 (510h36m24.98892626s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:26.829253118 (510h36m26.700336153s) [upload=448065]: Next returned nil segment Dec 26 22:05:26.829266823 (510h36m26.700349868s) [upload=448065]: Waiting for error group managing segments... Dec 26 22:05:27.114702552 (510h36m26.985785587s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.117900832 (510h36m26.988983867s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.135399732 (510h36m27.006482767s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137100431 (510h36m27.008183476s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137226731 (510h36m27.008309776s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.143949792 (510h36m27.015032827s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.155016275 (510h36m27.02609932s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.168724899 (510h36m27.039807934s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.298348504 (510h36m27.169431549s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.338925720 (510h36m27.210008765s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.346580496 (510h36m27.217663531s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.426919254 (510h36m27.298002289s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460645785 (510h36m27.331728841s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460670171 (510h36m27.331753206s) [seg_pos={0 0}, upload=448065]: Segment reached optimal threshold of 13 pieces. Dec 26 22:05:27.460905920 (510h36m27.331988965s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection Dec 26 22:05:27.460913555 (510h36m27.3319966s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461068510 (510h36m27.332151565s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 22:05:27.461079521 (510h36m27.332162576s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461130388 (510h36m27.332213433s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection Dec 26 22:05:27.461138514 (510h36m27.332221549s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461255155 (510h36m27.3323382s) [seg_pos={0 0}, upload=448065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:05:27.461370555 (510h36m27.33245359s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:05:27.461377589 (510h36m27.332460624s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:05:27.758634566 (510h36m27.629717601s) [upload=448065]: Done uploading segments... Dec 26 22:05:27.758653101 (510h36m27.629736136s) [upload=448065]: Upload finished. err: <nil> Duration: 6.2368828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX.X......X..X...X..XX.X...XX.........X.................X.] duration: 7.543516963s trace: https://jaeger.dev.storj.io/trace/b82db2838f61e63 FAIL (2025-12-26 22:06:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 22:05:24.864904308 (510h36m24.735987353s) [upload=448065]: Got next segment. Inline: false Dec 26 22:05:24.864952599 (510h36m24.736035654s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:05:24.864959422 (510h36m24.736042467s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:05:25.111861816 (510h36m24.982944861s) [seg_pos={0 0}, upload=448065]: Begin upload segment... Dec 26 22:05:25.111935336 (510h36m24.983018381s) [seg_pos={0 0}, upload=448065]: Done begin upload segment. Dec 26 22:05:25.111966516 (510h36m24.983049551s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.111991903 (510h36m24.983074938s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.112535639 (510h36m24.983618684s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113043667 (510h36m24.984126702s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113582021 (510h36m24.984665066s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114083656 (510h36m24.985166691s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114396211 (510h36m24.985479246s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114719127 (510h36m24.985802182s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115110252 (510h36m24.986193287s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115356320 (510h36m24.986439365s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116335635 (510h36m24.98741868s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116610929 (510h36m24.987693974s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116696833 (510h36m24.987779898s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117284651 (510h36m24.988367697s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117487418 (510h36m24.988570463s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117843225 (510h36m24.98892626s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:26.829253118 (510h36m26.700336153s) [upload=448065]: Next returned nil segment Dec 26 22:05:26.829266823 (510h36m26.700349868s) [upload=448065]: Waiting for error group managing segments... Dec 26 22:05:27.114702552 (510h36m26.985785587s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.117900832 (510h36m26.988983867s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.135399732 (510h36m27.006482767s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137100431 (510h36m27.008183476s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137226731 (510h36m27.008309776s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.143949792 (510h36m27.015032827s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.155016275 (510h36m27.02609932s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.168724899 (510h36m27.039807934s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.298348504 (510h36m27.169431549s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.338925720 (510h36m27.210008765s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.346580496 (510h36m27.217663531s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.426919254 (510h36m27.298002289s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460645785 (510h36m27.331728841s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460670171 (510h36m27.331753206s) [seg_pos={0 0}, upload=448065]: Segment reached optimal threshold of 13 pieces. Dec 26 22:05:27.460905920 (510h36m27.331988965s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection Dec 26 22:05:27.460913555 (510h36m27.3319966s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461068510 (510h36m27.332151565s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 22:05:27.461079521 (510h36m27.332162576s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461130388 (510h36m27.332213433s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection Dec 26 22:05:27.461138514 (510h36m27.332221549s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461255155 (510h36m27.3323382s) [seg_pos={0 0}, upload=448065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:05:27.461370555 (510h36m27.33245359s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:05:27.461377589 (510h36m27.332460624s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:05:27.758634566 (510h36m27.629717601s) [upload=448065]: Done uploading segments... Dec 26 22:05:27.758653101 (510h36m27.629736136s) [upload=448065]: Upload finished. err: <nil> Duration: 7.543548483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........X...................................] duration: 9.607840132s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:06:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 9.607862285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...XX.X......X..X...X..XX.X...XX.........X.................X] duration: 30.002892074s trace: https://jaeger.dev.storj.io/trace/b82db2838f61e63 FAIL (2025-12-26 22:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== g segments... Dec 26 22:05:24.864904308 (510h36m24.735987353s) [upload=448065]: Got next segment. Inline: false Dec 26 22:05:24.864952599 (510h36m24.736035654s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 22:05:24.864959422 (510h36m24.736042467s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 22:05:25.111861816 (510h36m24.982944861s) [seg_pos={0 0}, upload=448065]: Begin upload segment... Dec 26 22:05:25.111935336 (510h36m24.983018381s) [seg_pos={0 0}, upload=448065]: Done begin upload segment. Dec 26 22:05:25.111966516 (510h36m24.983049551s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.111991903 (510h36m24.983074938s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.112535639 (510h36m24.983618684s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113043667 (510h36m24.984126702s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.113582021 (510h36m24.984665066s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114083656 (510h36m24.985166691s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114396211 (510h36m24.985479246s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.114719127 (510h36m24.985802182s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115110252 (510h36m24.986193287s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.115356320 (510h36m24.986439365s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116335635 (510h36m24.98741868s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116610929 (510h36m24.987693974s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.116696833 (510h36m24.987779898s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117284651 (510h36m24.988367697s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117487418 (510h36m24.988570463s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:25.117843225 (510h36m24.98892626s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Uploading piece... Dec 26 22:05:26.829253118 (510h36m26.700336153s) [upload=448065]: Next returned nil segment Dec 26 22:05:26.829266823 (510h36m26.700349868s) [upload=448065]: Waiting for error group managing segments... Dec 26 22:05:27.114702552 (510h36m26.985785587s) [noise=true , address=145.79.227.41:4087 , piece_id=3WCM2Y7Q3Z6ZFHUQYXAGYCYEJBRNYQ5EMTM5G263P5PEUSDKIYNQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.117900832 (510h36m26.988983867s) [noise=true , address=145.79.227.36:4518 , piece_id=EOUXT3GOF5KCVF64HM2UC2OL4HCMRZVVMICJDHO2R2LTNLGZ4E7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.135399732 (510h36m27.006482767s) [noise=true , address=145.79.227.48:4303 , piece_id=LMIUVPDU4QOY3IOBQLBCYH5KPT4CPNOCGGOY7PBABGDUHFMBAXOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137100431 (510h36m27.008183476s) [noise=true , address=145.79.227.29:4942 , piece_id=NHHNBO2PWJKBGTXI75HYDSLH47FZP54AVS3ACMZ7OG2RDGNOJ2PQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.137226731 (510h36m27.008309776s) [noise=true , address=145.79.227.49:4339 , piece_id=NAJKLGBRCURYKM2TAZT26FJSBOQDXNM63UVYPONRJO46Y4776KEA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.143949792 (510h36m27.015032827s) [noise=true , address=145.79.227.11:4729 , piece_id=WSEQQFLP3U3NTVY4PZ2WWY4KDMML4ACMIJMT65SB43WHEJMD2IDQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.155016275 (510h36m27.02609932s) [noise=true , address=145.79.227.39:4013 , piece_id=OW5F5PTNYDQ6MKI4OLYFC236ZXWIX7VKJGC4DB6CKAKEXK5PTQKA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.168724899 (510h36m27.039807934s) [noise=true , address=145.79.227.47:4265 , piece_id=B4S74B6XFANYYNUSVRLKDZH7S66F3XKUJ7K7POSL63FIJB36TTRQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.298348504 (510h36m27.169431549s) [noise=true , address=45.91.116.8:4048 , piece_id=YRMCFV4RZL4DZTI775O4RIHYRT5M7X5GR27PC44DLV2FRCWCIL7A, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.338925720 (510h36m27.210008765s) [noise=true , address=45.91.116.5:4297 , piece_id=3T5UN7PHV3ZF3EFZ3Z64GH6JBTUQ2IO3F77WJRU6KIF54C3P5SAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.346580496 (510h36m27.217663531s) [noise=true , address=45.91.116.7:4009 , piece_id=RYO54BIFWMXBR4PP45B3PXJDODCKLQGEU6URWZJCWMJWA42O4RGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.426919254 (510h36m27.298002289s) [noise=true , address=45.91.116.11:4120 , piece_id=VITB5DPLUTEKVHBWW7FXEWELYR45HFRJVKENDZBD6XEZA6PZ3WOA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460645785 (510h36m27.331728841s) [noise=true , address=45.91.116.13:4154 , piece_id=E72DVCHVSVFRWPS6J7ILS6P2WHD4QOKXLCVSPU7WGLRHDQGBRGCA, seg_pos={0 0}, upload=448065]: Done uploading piece. err: <nil> Dec 26 22:05:27.460670171 (510h36m27.331753206s) [seg_pos={0 0}, upload=448065]: Segment reached optimal threshold of 13 pieces. Dec 26 22:05:27.460905920 (510h36m27.331988965s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF): protocol: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41964->145.79.227.6:4258: use of closed network connection Dec 26 22:05:27.460913555 (510h36m27.3319966s) [noise=true , address=145.79.227.6:4258 , piece_id=OXLJNRV7TUQC7P237J4OLVKPYOGQ3JIPGDUOP6RGEDM2LCYB6MAQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461068510 (510h36m27.332151565s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 22:05:27.461079521 (510h36m27.332162576s) [noise=true , address=45.91.116.9:4263 , piece_id=DWNCSYEOLFN4JHQUZQ5F4RTI7HOINHH6O7OCZ4YVI4LSGS7CDFGQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461130388 (510h36m27.332213433s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50194->77.90.31.13:4627: use of closed network connection Dec 26 22:05:27.461138514 (510h36m27.332221549s) [noise=true , address=77.90.31.13:4627 , piece_id=ZUUEQNZDHER4MQ5OJUJJNJOAXYIUG4J7252GUY5G73A4I3EFDICQ, seg_pos={0 0}, upload=448065]: Long tail cancellation due to optimal threshold Dec 26 22:05:27.461255155 (510h36m27.3323382s) [seg_pos={0 0}, upload=448065]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 22:05:27.461370555 (510h36m27.33245359s) [upload=448065]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 22:05:27.461377589 (510h36m27.332460624s) [upload=448065]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 22:05:27.758634566 (510h36m27.629717601s) [upload=448065]: Done uploading segments... Dec 26 22:05:27.758653101 (510h36m27.629736136s) [upload=448065]: Upload finished. err: <nil> Duration: 30.002909588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........X..................................] duration: 7.624743021s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:05:51 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 7.624767468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...XX.X......X..X...X..XX.X...XX.........X.................] duration: 16.328916787s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:05:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 16.328947405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.........X.................................] duration: 11.434095268s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:05:02 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 11.434128321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........X................................] duration: 22.560269758s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:04:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 22.560294786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X...XX.X......X..X...X..XX.X...XX.........X................] duration: 6.098356033s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:04:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.098387434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.........X...............................] duration: 7.7618205s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:03:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 7.761854966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...XX.X......X..X...X..XX.X...XX.........X...............] duration: 9.377331974s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:03:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 9.377359527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.........X..............................] duration: 6.585618885s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:03:32 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 6.585647831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X...XX.X......X..X...X..XX.X...XX.........X..............] duration: 9.388125557s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:02:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 9.388152197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.........X.............................] duration: 5.96685303s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:02:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 5.966899167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.........X............................] duration: 12.715567868s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:02:35 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 12.715586793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.X...XX.X......X..X...X..XX.X...XX.........X.............] duration: 7.460129419s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:02:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 7.460153183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.........X...........................] duration: 16.299590655s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:02:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 16.299605473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.........X..........................] duration: 15.160413431s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:01:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 15.160443477s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X.X...XX.X......X..X...X..XX.X...XX.........X............] duration: 6.128509891s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:01:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.12853623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X.X...XX.X......X..X...X..XX.X...XX.........X...........] duration: 6.152602768s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 22:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.152631702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.........X.........................] duration: 11.673581979s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 22:00:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 11.673618278s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X..X.X...XX.X......X..X...X..XX.X...XX.........X..........] duration: 6.334640359s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:59:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.334667369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.........X........................] duration: 6.407180119s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:59:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 6.407208401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X.........] duration: 6.292079195s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:59:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.292106236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X........] duration: 28.264704563s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:58:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 28.264733968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.........X.......................] duration: 10.045698235s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:58:12 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 10.045724015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.........X......................] duration: 15.323622158s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:57:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 15.323647105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X.......] duration: 6.024248021s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:57:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.024271154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.........X.....................] duration: 6.052056382s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:56:45 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 6.05209717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X......] duration: 5.974685685s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 5.974712266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.........X....................] duration: 7.390349095s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:56:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 7.39036671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X.....] duration: 6.076678997s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:55:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.076711218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.........X...................] duration: 17.088695746s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:55:37 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 17.088721204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.........X..................] duration: 5.748842092s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:55:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 5.748867289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X....] duration: 6.028226374s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:54:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.028251092s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.........X.................] duration: 9.058495064s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:54:13 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 9.058522347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.........X................] duration: 16.997232796s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:54:00 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 16.997249127s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X...] duration: 6.333095608s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:53:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 6.333122069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.........X...............] duration: 27.213285538s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:53:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 27.213313169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X..] duration: 25.917218174s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:53:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 25.917244966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....XX.XX.X..X.X...XX.X......X..X...X..XX.X...XX.........X.] duration: 7.29965617s trace: https://jaeger.dev.storj.io/trace/77da6b73901445a8 FAIL (2025-12-26 21:52:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA) context deadline exceeded ======LOGS====== :51:10.888895178 (510h22m10.759978223s) [upload=447858]: Starting upload Dec 26 21:51:10.888961785 (510h22m10.76004483s) [upload=447858]: Done starting upload Dec 26 21:51:10.889308644 (510h22m10.760391679s) [upload=447858]: Uploading segments... Dec 26 21:51:10.889362778 (510h22m10.760445833s) [upload=447858]: Got next segment. Inline: false Dec 26 21:51:10.889401812 (510h22m10.760484847s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:51:10.889407222 (510h22m10.760490267s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:51:11.135820935 (510h22m11.006903991s) [seg_pos={0 0}, upload=447858]: Begin upload segment... Dec 26 21:51:11.135883093 (510h22m11.006966128s) [seg_pos={0 0}, upload=447858]: Done begin upload segment. Dec 26 21:51:11.135916547 (510h22m11.006999592s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136050392 (510h22m11.007133447s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.136667587 (510h22m11.007750622s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137111032 (510h22m11.008194077s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.137583521 (510h22m11.008666566s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.138250842 (510h22m11.009333887s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139054662 (510h22m11.010137707s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.139647321 (510h22m11.010730366s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.140322968 (510h22m11.011406013s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.141704328 (510h22m11.012787374s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.142652614 (510h22m11.013735659s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144030337 (510h22m11.015113372s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.144562371 (510h22m11.015645416s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146262198 (510h22m11.017345243s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.146668270 (510h22m11.017751315s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:11.148153719 (510h22m11.019236784s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Uploading piece... Dec 26 21:51:12.853423810 (510h22m12.724506855s) [upload=447858]: Next returned nil segment Dec 26 21:51:12.853440461 (510h22m12.724523506s) [upload=447858]: Waiting for error group managing segments... Dec 26 21:51:13.140103005 (510h22m13.01118606s) [noise=true , address=145.79.227.9:4365 , piece_id=6VNF7RQQAHSW37CJYKW7UBD57R2QYCBJHF5SD46H6VBAVYMNCVYA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.142984432 (510h22m13.014067477s) [noise=true , address=145.79.227.46:4603 , piece_id=7W267IFQYAUGXIWOAEIKISNRYXVBZUXCEJEDWSQXETNXMH2NMSKQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.151475841 (510h22m13.022558886s) [noise=true , address=145.79.227.49:4338 , piece_id=B5ODUI5PI3RKS2RRLIGXY3LN6ED2EEYCHNMCHQHVNDCA6GCPBXPQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.152957492 (510h22m13.024040527s) [noise=true , address=145.79.227.45:4230 , piece_id=MDI4LINUS5UWPSXFG5RKN4NCUM3BQ4QIEPQWQ7J6GTSBILJJRISQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.159593397 (510h22m13.030676432s) [noise=true , address=145.79.227.13:4802 , piece_id=5AR5RYVDVJPOKLOOSR4G2H3E4PSNA75XNCCDYAUWV46AQO2WJ7JQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.160996830 (510h22m13.032079865s) [noise=true , address=145.79.227.7:4293 , piece_id=FOT3T7GHFU2VDSY3UBTJ3CKY6HERINXYKL3EUA6UFFA44FX6REVQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.163154599 (510h22m13.034237634s) [noise=true , address=145.79.227.17:4972 , piece_id=3WHHW7P23N7BZ34PE2Q6O3PQA2ZDRFTYCCJZAXIRUWONGJZJSYIQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.166380100 (510h22m13.037463135s) [noise=true , address=145.79.227.51:4409 , piece_id=QMXZLAWT2MCBVFH2Q3UWDEAANXYTY6EMMCMIEHOWINUMSFNGCWEQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.169247721 (510h22m13.040330756s) [noise=true , address=145.79.227.47:4265 , piece_id=TSFWMFSJTLX766LVB7MT3GKYQOYWSOQGUA3QAHD6VQCWAX6P3PTA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.171349032 (510h22m13.042432077s) [noise=true , address=145.79.227.52:4446 , piece_id=NAVNTFVIL2UB5P3IS4ZBOWFAI2RD2YESD33FDK4NZWHFQC3WF35Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.344875406 (510h22m13.215958441s) [noise=true , address=45.91.116.12:4226 , piece_id=FKRJKFSVEUWCUNHV4VPPPCK7CVOK2OBS45XHRBN2OOFFAHLN3ZCA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.391222238 (510h22m13.262305273s) [noise=true , address=45.91.116.14:4190 , piece_id=K5AO3UW4EQXOENYQ34ZFGDJLRDXR7LNYMGKDHSBLKMXEFQD4QY5A, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415678176 (510h22m13.286761221s) [noise=true , address=45.91.116.11:4118 , piece_id=W54Z5YLXNN3QB5ZGPF4N4R46QDCE32RTDO3AT4VIEH22ZJQA4ZMA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: <nil> Dec 26 21:51:13.415713514 (510h22m13.286796549s) [seg_pos={0 0}, upload=447858]: Segment reached optimal threshold of 13 pieces. Dec 26 21:51:13.415908053 (510h22m13.286991099s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41988->145.79.227.19:4079: use of closed network connection Dec 26 21:51:13.415917913 (510h22m13.287000958s) [noise=true , address=145.79.227.19:4079 , piece_id=QUPRNB773HAB3QSLDSCIBR2YKE4J3HLFFGAFPF4JSMEHTS7WHW6Q, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.415997474 (510h22m13.287080519s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416013053 (510h22m13.287096088s) [noise=true , address=77.90.31.10:4520 , piece_id=W3WCX3QMR64D4ROG2PXJLWIATND76JXJFIBN6KWJRFHU462J6BRQ, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416174962 (510h22m13.287257997s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 26 21:51:13.416182055 (510h22m13.28726509s) [noise=true , address=77.90.31.19:4842 , piece_id=P2D42H526DLHUXVFITMQ6WW7INU6KTCEMHYT45WCPQVJPQ2KV5DA, seg_pos={0 0}, upload=447858]: Long tail cancellation due to optimal threshold Dec 26 21:51:13.416200661 (510h22m13.287283696s) [seg_pos={0 0}, upload=447858]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:51:13.416266376 (510h22m13.287349422s) [upload=447858]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:51:13.416270975 (510h22m13.28735401s) [upload=447858]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:51:13.723410159 (510h22m13.594493194s) [upload=447858]: Done uploading segments... Dec 26 21:51:13.723427653 (510h22m13.594510688s) [upload=447858]: Upload finished. err: <nil> Duration: 7.299682761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.........X..............] duration: 14.358656083s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:52:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 14.358683164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.........X.............] duration: 5.893507379s trace: https://jaeger.dev.storj.io/trace/43b1de88cbc0f7e8 FAIL (2025-12-26 21:51:54 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791034323 (510h14m14.662117368s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791411239 (510h14m14.662494284s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791475853 (510h14m14.662558888s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.791806903 (510h14m14.662889959s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792204971 (510h14m14.663288016s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792359596 (510h14m14.663442641s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.792599403 (510h14m14.663682438s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793036875 (510h14m14.66411992s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.793771604 (510h14m14.664854639s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794556199 (510h14m14.665639234s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.794854256 (510h14m14.665937301s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.795332016 (510h14m14.666415061s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796137539 (510h14m14.667220584s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:14.796624166 (510h14m14.667707201s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Uploading piece... Dec 26 21:43:16.264912523 (510h14m16.135995558s) [upload=447735]: Next returned nil segment Dec 26 21:43:16.264930898 (510h14m16.136013933s) [upload=447735]: Waiting for error group managing segments... Dec 26 21:43:16.650857309 (510h14m16.521940354s) [noise=true , address=145.79.227.35:4489 , piece_id=HDYI7VSIMPMBERJBBP22SGSVEXZAVGA4E3HHLBIEPQSLWHTHQS2A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.668025210 (510h14m16.539108245s) [noise=true , address=145.79.227.16:4938 , piece_id=3B6XRVRGA5FR7DI7AXFDCTIIRTATEWWGFRP6K2STNGSGCGAS36SA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.700914036 (510h14m16.571997071s) [noise=true , address=145.79.227.45:4249 , piece_id=23M7P7ZMQJUAMZJI4HXBE3QFBIIXXJYDBWOCC4NVFZMYSIU2JGJA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.702614003 (510h14m16.573697038s) [noise=true , address=145.79.227.38:4990 , piece_id=CFS4CSHS6UYB4EB7NGWHGAWHQU342CBOJUHTXNVDZTGZSWOC5S5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.703508877 (510h14m16.574591912s) [noise=true , address=145.79.227.10:4404 , piece_id=GFQGZXJXPDDD2CWQXV7YRMRGDJJ3YOTYXY2UYRMGADBU7UYAVGXQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.865709536 (510h14m16.736792571s) [noise=true , address=77.90.31.11:4566 , piece_id=VBJXU2EUBYLMUM2ZEJA5DAQX5WOGYZNAYNNUHUOI3DIUV2OVOJKQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.950099629 (510h14m16.821182674s) [noise=true , address=77.90.31.10:4549 , piece_id=BD5EP6CGLGD4HO45PDVE3AXEOHP57OM5UPHC3CHIVXN4UQ7U23GA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.951888615 (510h14m16.82297165s) [noise=true , address=45.91.116.9:4294 , piece_id=DWDRDE46JCF64OJBUS2RSSJTB7IL777AODLT7GJCMHO7XW5XFIGA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:16.971372987 (510h14m16.842456032s) [noise=true , address=77.90.31.17:4802 , piece_id=6BVDOUYYKL5W2CIJD3FNKZDYLG6DTC4G32ADXIODWCKM6GHFBWFA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.065428024 (510h14m16.936511059s) [noise=true , address=77.90.31.16:4747 , piece_id=VDW75O5K5H5OK6AZH5LVFPSD57JLSM4XFJVGOSRNHZGQ2G3CGB3A, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.224391518 (510h14m17.095474583s) [noise=true , address=45.91.116.11:4144 , piece_id=CCJKNVQICDUV77N4PGI2DJ6VLQ7FPPKMC2X3MFBVQ5VPQRFI6YRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.234254889 (510h14m17.105337924s) [noise=true , address=45.91.116.8:4069 , piece_id=INRI4IGETIM4EPVW3WG5GKNDES6H3PEJWAO6EGV2666JEFO36N6Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.556192535 (510h14m17.42727557s) [noise=true , address=77.90.31.13:4643 , piece_id=JNNYWKQKIP27CQQ7V2EFABRGOAED72ORLJQEBVTN3HIPPTGB7ISQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586071319 (510h14m17.457154364s) [noise=true , address=145.79.227.44:4196 , piece_id=KQVMH2GCAMCJHIA2O5IYBHEMHYV5HMI6XCVKAAPMVDPAJWGLIH2Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: <nil> Dec 26 21:43:17.586094724 (510h14m17.457177759s) [seg_pos={0 0}, upload=447735]: Segment reached optimal threshold of 14 pieces. Dec 26 21:43:17.586291839 (510h14m17.457374874s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ucSAZVBqRRJU7WZDKs7Cs6yKV73haX7px76sfukA6b8Q8DDsW): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586299965 (510h14m17.457383s) [noise=true , address=77.90.31.15:4702 , piece_id=Y355Q3B2XXZE7XVQH3PY2G2V4ZIRYNM53DTXANLMZ6HCROPSJMQA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586341193 (510h14m17.457424228s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19KofWeQV4jMQzmSCv3LUsUqAqGfAuDqyo9GGsqKGi7ctT3FA2): protocol: expected piece hash; context canceled; EOF Dec 26 21:43:17.586355821 (510h14m17.457438856s) [noise=true , address=145.79.227.40:4076 , piece_id=TOJHOGRFCXWML2ZBBVICFCKF3MXKURDWD6SCYK7Y73WPAHKH2BRA, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586497881 (510h14m17.457580916s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sNBX9Xaei33rZ1ukq9qUkco2xiEhVSP895V1M1XU3dwYsVsqD): protocol: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35564->45.91.116.5:4323: use of closed network connection Dec 26 21:43:17.586503563 (510h14m17.457586598s) [noise=true , address=45.91.116.5:4323 , piece_id=EZJWGRZGGXYLPEQLXUT4KNERQJPQ6JM63G7QGIVB57Z6X3QUWR5Q, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586544370 (510h14m17.457627425s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UrPGB76X7LaXe3ioiu4NtUQ5FwspKYtBWD6rcMmZC6xbNnCAF): protocol: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42926->145.79.227.17:4997: use of closed network connection Dec 26 21:43:17.586552656 (510h14m17.457635701s) [noise=true , address=145.79.227.17:4997 , piece_id=OSJDYXXS65RCQQ4FRSDBXTA2QVMIGI2IOADJ3MN2XQJB36UH65CQ, seg_pos={0 0}, upload=447735]: Long tail cancellation due to optimal threshold Dec 26 21:43:17.586571882 (510h14m17.457654927s) [seg_pos={0 0}, upload=447735]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 21:43:17.586639741 (510h14m17.457722776s) [upload=447735]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:43:17.586645373 (510h14m17.457728418s) [upload=447735]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:43:17.882447271 (510h14m17.753530326s) [upload=447735]: Done uploading segments... Dec 26 21:43:17.882471747 (510h14m17.753554792s) [upload=447735]: Upload finished. err: <nil> Duration: 5.893531234s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)